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1971-2013 
We miss you. 
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About the Dedication 


Malcolm Tredinnick wasnt just a Django core developer and reviewer of “Two Scoops of Django: 
Best Practices for Django 1.5.” To us, he was much, much more. 

Daniel had worked with Malcolm Tredinnick in the summer of 2010, but we first met him in person 
at DjangoCon 2010. He was funny and charming, sharply opinionated but always a gentleman; we 
instantly became close friends. 

In 2012, when we co-organized the first PyCon Philippines, as soon as we told him about it, Mal¬ 
colm instantly declared he was coming. He gave two memorable talks and ran an impromptu all-day 
Django tutorial. He also pushed and encouraged the local community to work on Filipino language 
translations for Django, including Tagalog, Tausug, Cebuano, and more. 

After the conference, we started working on a book about Django best practices. We gathered friends 
and colleagues to help us as technical reviewers. Malcolm Tredinnick became the most active of them. 
He was our mentor and forced us to dig deeper and work harder. He did this while working a day 
job as the leader of a combined Rails and Haskell team; Malcolm was a true programming language 
polyglot. 

For our book, he provided so much assistance and guidance we tried to figure out a way to include 
him in the author credits. When we told him about our dilemma, he laughed it olf saying, “For a 
book caUed ‘Two Scoops’, you caht have three authors.” We suggested he share credit with us on 
a second book, and he refused, saying he preferred to just comment on our work. He said that he 
wanted people to have proper references, and for him, simply reviewing our work was contributing 
to the greater good. Eventually the two of us quietly planned to somehow coerce him into being a 
co-author on a future work. 

After months of effort, we released the first iteration on January 17th, 2013. Malcolm stepped back 
from Two Scoops of Django, but we stayed in touch. Since Malcolm was unable to attend PyCon 
US 2013 we wereht sure when we would meet him again. 

Two months later, on March 17th, 2013, Malcolm passed away. 

We knew Malcolm for less than three years and yet he made an incredible difference in our lives. 
We’ve heard many similar stories in the community about Malcolm; He was a friend and mentor to 
countless others around the world. His last lesson to us went beyond code or writing, he taught us 
to never take for granted friends, family, mentors, and teachers. 
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A Few Words From Daniel Roy Greenfeld 

In the spring of2006,1 was working for NASA on a project that implemented a Java-based RESTful 
web Service that was taking weeks to deliver. One evening, when management had left for the day, I 
reimplemented the Service in Python in 90 minutes. 

I knew then that I wanted to work with Python. 

I wanted to use Django for the web front-end of the web Service, but management insisted on using 
a closed-source stack because “Django is only at version 0.9x, hence not ready for real projects.” I 
disagreed, but stayed happy with the realization that at least the core architecture was in Python. 
Django used to be edgy during those heady days, and it scared people the same way that Node.js 
scares people today. 

Over ten years later, Django is considered a mature, powerful, secure, stable framework used by 
incredibly successful corporations (Instagram, Pinterest, MoziUa, etc.) and government agencies 
(NASA, Library of Congress, et al) all over the world. Convincing management to use Django isnt 
hard anymore, and if it is hard to convince them, finding jobs which let you use Django has become 
much easier. In my many years of building Django projects, Ive learned how to launch new web 
applications with incredible speed while keeping technical debt to an absolute minimum. 

My goal in this book is to share with you what IVe learned. My knowledge and experience have been 
gathered from advice given by core developers, mistakes IVe made, successes shared with others, and 
an enormous amount of note taking. Pm going to admit that the book is opinionated, but many of 
the leaders in the Django community use the same or similar techniques. 

This book is for you, the developers. I hope you enjoy it! 
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A Few Words From Audrey Roy Greenfeld 

I first discovered Python in a graduate class at MIT in 2005. In less than 4 weeks of homework 
assignments, each student built a voice-controlled system for navigating between rooms in MIT’s 
Stata Center, running on our HP iPaqs running Debian. I was in awe of Python and wondered why 
it wasnt used for everything. I tried building a web application with Zope but struggled with it. 

A couple of years passed, and I got drawn into the Silicon Valley tech startup scene. I wrote graphics 
libraries in C and desktop applications in C++ for a startup. At some point, I left that job and picked 
up painting and sculpture. Soon I was drawing and painting frantically for art shows, co-directing a 
140-person art show, and managing a series of real estate renovations. I realized that I was doing a 
lot at once and had to optimize. Naturally, I turned to Python and began writing Scripts to generate 
some of my artwork. That was when I rediscovered the joy of working with Python. 

Many friends from the Google App Engine, SuperHappyDevHouse, and hackathon scenes in Silicon 
Valley inspired me to get into Django. Through them and through various freelance projects and 
partnerships I discovered how powerful Django was. 

Before I knew it, I was attending PyCon 2010, where I met my husband Daniel Roy Greenfeld. We 
met at the end of James Bennett’s “Django In Depth” tutorial, and now this chapter in our lives has 
come fuU circle with the publication of this book. 

Django has brought more joy to my life than I thought was possible with a web framework. My goal 
with this book is to give you the thoughtful guidance on common Django development practices that 
are normally left unwritten (or implied), so that you can get past common hurdles and experience 
the joy of using the Django web framework for your projects. 
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Introduction 


Our aim in writing this book is to write down ali of the unwritten tips, tricks, and common practices 
that weVe learned over the years while working with Django. 

While writing, weVe thought of ourselves as scribes, taking the various things that people assume 
are common knowledge and recording them with simple examples. 


A Word About Our Recommendations 

Like the oflicial Django documentation, this book covers how to do things in Django, illustrating 
various scenarios with code examples. 

Unlike the Django documentation, this book recommends particular coding styles, patterns, and 
library choices. While core Django developers may agree with some or many of these choices, keep 
in mind that many of our recommendations are just that: personal recommendations formed after 
years of working with Django. 

Throughout this book, we advocate certain practices and techniques that we considet to be the best 
approaches. We also express our own personal preferences for particular tools and libraries. 

Sometimes we reject common practices that we considet to be anti-patterns. For most things we 
reject, we try to be polite and respectful of the hard work of the authors. There are the rare, few 
things that we may not be so polite about. This is in the interest of helping you avoid dangerous 
pitfalls. 

We have made every effort to give thoughtful recommendations and to make sure that our practices 
are sound. WeVe subjected ourselves to harsh, nerve-wracking critiques from Django and Python 
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core developers whom we greatly respect. WeVe had this book reviewed by more technical reviewers 
than the average technical book, and weVe poured countless hours into revisions. That being said, 
there is always the possibility of errors or omissions. There is also the possibility that better practices 
may emerge than those described here. 

We are fuUy committed to iterating on and improving this book, and we mean it. If you see any 
practices that you disagree with or anything that can be done better, we humbly ask that you send us 
your suggestions for improvements. The best way to send us feedback is to file an issue at 
github.com/twoscoops/two-scoops-of-dj ango-1.11/1ssues. 

Please don’t hesitate to teli us what can be improved. We will take your feedback constructively. Er¬ 
rata will be published at https://glthub.com/twoscoops/two-scoops-of-django-1.11/ 
blob/master/errata.md. 


WhyTwo Scoops of Django? 

Like most people, we, the authors ofthis book, love ice cream. Every Saturday nightwe throwcaution 
to the wind and indulge in ice cream. Dont teli anyone, but sometimes we even have some when it’s 
not Saturday night! 



Figure 1: Throwing caution to the wind. 


We like to try new flavors and discuss their merits against our old favorites. Tracking our progress 









through ali these flavors, and possibly building a club around it, makes for a great sample Django 
project. 

When we do find a flavor we really like, the new flavor brings a smile to our face, just like when we 
find great tidbits of code or advice in a technical book. One of our goals for this book is to write the 
kind of technical book that brings the ice cream smile to readers. 

Best of all, using ice cream analogies has allowed us to come up with more vivid code examples. 
WeVe had a lot of fun writing this book. You may see us go overboard with ice cream silliness here 
and there; please forgive us. 


Before You Begin 

This book is not a tutorial. If you are new to Django, this book will be helpful but large parts will 
be challenging for you. To use this book to its fuUest extent, you should have an understanding of 
the Python programming language and have at least gone through the entire multi-page Django 
tutorial: docs.djangoproject.eom/en/l.ll/intro/tutorial01/. Experience with object- 
oriented programming is also very useful. 


This Book Is Intended for Django 1.11 and Python 3.6 jc/2.7.12 

This book should work weU with the Django 1.11 series, less so with Django 1.10, and so on. Even 
though we make no promises about functional compatibility, at least the general approaches from 
most of this book stand up over every post-1.0 version of Django. 

As for the Python version, this book is tested on Python 3.6 and Python 2.7.13. 

While the book is tested on Python 3.6, we avoided the use of f-stri ngli ter ais. While we enjoy 
using f-strings, we wanted to ensure that our code examples worked on earlier versions of Python. 
The same goes for the use of Python 3’s simplified syntax for the super () built-in. 


Each Chapter Stands on Its Own 

Unlike tutorial and walkthrough books where each chapter builds upon the previous chapters project, 
weVe written this book in a way that each chapter intentionaUy stands by itself 
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WeVe done this in order to make it easy for you to reference chapters about specific topics when 
needed while youre working on a project. 

The examples in each chapter are completely independent. They arent intended to be combined into 
one project and are not a tutorial. Consider them useful, isolated snippets that illustrate and help 
with various coding scenarios. 


Conventions Used in This Book 

Code examples like the foUowing are used throughout the book: 



To keep these snippets compact, we sometimes violate the PEP 8 conventions on comments and line 

spacing. Code samples are available online at 

github.com/twoscoops/two-scoops-of-dj ango-1.11. 

Special “Don’t Do This!” code blocks like the following indicate examples of bad code that you should 
avoid: 


Example 2: “Don’t Do This!” Code Example 

class Scoop: 

def _init_(self): 

self._is_yummy = False 


We use the following typographical conventions throughout the book: 

>- Constant widthor text Shaded constant width for code fragments or commands. 
>- Italic for filenames. 

>- Bold when introducing a new term or important word. 

Boxes containing notes, warnings, tips, and little anecdotes are also used in this book: 










TIP: Something You Should Know 


Tip boxes give handy advice. 


WARNING: Some Dangerous Pitfall 


Warning boxes help you avoid common mistakes and pitfalls. 


PACKAGE TIP: Some Usefiil Package Recommendation 


Indicates notes about useful third-party packages related to the current chapter, and general 
notes about using various Python, Django, and front-end packages. 

We also provide a complete list of packages recommended throughout the book in Appendix 
A: Packages Mentioned In This Book. 


We also use tables to summarize Information in a handy, concise way: 



Daniel Roy Greenfeld 

Audrey Roy Greenfeld 

Can be fed coconut ice cream 

No 

Yes 

Favorite ice cream flavors of the moment 

Pumpkin 

Mint Chocolate Chip 


Authors’ Ice Cream Preferences 


Core Concepts 

When we build Django projects, we keep the foUowing concepts in mind. 


Keep It Simple, Stupid 


Kelly Johnson, one of the most renowned and prolific aircraft design engineers in the history of 
aviation, said it this way about 50 years ago. Centuries earlier, Leonardo da Vinci meant the same 
thing when he said, “Simplicity is the ultimate sophistication.” 
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When building Software projects, each piece of unnecessary complexity makes it harder to add new 
features and maintain old ones. Attempt the simplest solution, but take care not to implement overly 
simplistic Solutions that make bad assumptions. This concept is sometimes abbreviated as “KISS.” 

Fat Models, Utility Modules, Thin Views, Stupid Templates 

When deciding where to put a piece of code, we like to follow the “Fat Models, Utility Modules, 
Thin Views, Stupid Templates” approach. 

We recommend that you err on the side of putting more logic into anything but views and templates. 
The results are pleasing. The code becomes clearer, more self-documenting, less duplicated, and a lot 
more reusable. As for template tags and filters, they should contain the least amount of logic possible 
to function. 

We cover this further in: 

Fat Models Section 6.7: Understanding Fat Models 
Utility Modules Section 29.2: Optimize Apps With Utility Modules 
Thin Views Section 8.5: Try to Keep Business Logic Out of Views 
Stupid Templates I Section 13.9: Follow a Minimalist Approach 
Stupid Templates II Chapter 14: Template Tags and Filters 

Start With Django by Default 

Before we consider switching out core Django components for things like alternative template en- 
gines, different ORMs, or non-relational databases, we first try an implementation using Standard 
Django components. Ifwe run into obstacles, we explore aU possibilities before replacing core Django 
components. 

See Chapter 18: Tradeoffs of Replacing Core Components. 

Be Familiarwith Django’s Design Philosophies 

It is good to periodically read the documentation on Django’s design philosophy because it helps 
us understand why Django provides certain constraints and tools. Like any framework, Django is 
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more than just a tool for providing views, it’s a way of doing things designed to help us put together 
maintainable projects in a reasonable amount of time. 

Reference docs.dj angoproj ect.com/en/1.ll/misc/design-philosophies/ 


The Twelve-Factor App 

A comprehensive approach to web-based application design, the Twelve-Factor App approach is grow- 
ing in popularity amongst many senior and core Django developers. It is a methodology for build- 
ing deployable, scalable applications worth reading and understanding. Parts of it closely match the 
practices espoused in Two Scoops of Django, and we like to think of it as suggested reading for any 
web-based application developer. 

See 12factor.net 


Our Writing Concepts 

When we wrote this book, we wanted to provide to the reader and ourselves the absolute best material 
possible. To do that, we adopted the foUowing principies: 


Provide the Best Material 

WeVe done our absolute best to provide the best material possible, going to the known resources 
on every topic covered to vet our material. We werent afraid to ask questions! Then we distilled the 
articles, responses and advice of experts into the content that exists in the book today. When that 
didn’t suffice, we came up with our own Solutions and vetted them with various subject matter experts. 
It has been a lot of work, and we hope you are pleased with the results. 

If you are curious about the differences between this edition (Django 1.11) and the previous edition 
(Django 1.8) of the book, you can find the shortlist of changes at 

github.com/twoscoops/two-scoops-of-dj ango-1.11/blob/master/changelog.md 
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Stand on the Shoulders of Giants 

While we take credit and responsibility for our work, we certainly did not come up with all the 
practices described in this book on our own. 

Without all of the talented, Creative, and generous developers who make up the Django, Python, and 
general open source Software communities, this book would not exist. We strongly believe in recog- 
nizing the people who have served as our teachers and mentors as well as our sources for information, 
and weVe tried our best to give credit whenever credit is due. 


Listen to Our Readers and Reviewers 

In the previous editions of this book, we received a huge amount of feedback from a veritable legion 
of readers and reviewers. This allowed us to greatly improve the quality of the book. It is now at a 
level that we hoped for but never expected to achieve. 

In return, we ve shared credit at the back of the book and are continually working on ways to pay it 
forward by improving the lives of developers around the world. 

If you have any questions, comments, or other feedback about this edition, please share your input 
by submitting issues in our issue tracker, at: 

>- github.com/twoscoops/two-scoops-of-dj ango-l.ll/i ssues 

Also, at the end of the book is a link to leave a review for Two Scoops of Django on /Vmazon. Doing 
this wiU help others make an informed decision about whether this book is right for them. 


Publish Issues and Errata 

Nothing is perfect, even after extensive review cycles. We will be publishing issues and errata at the 
Two Scoops of Django 1.11 GitHub repo: 

>- github.com/twoscoops/two-scoops-of-django-l. 11 
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Coding Style 


A little attention to following Standard coding style guidelines will go a long way. We highly recom- 
mend that you read this chapter, even though you may be tempted to skip it. 

1.1 The Importance of Making Your Code Readable 

Code is read more than it is written. An individual block of code takes moments to write, minutes 
or hours to debug, and can last forever without being touched again. It’s when you or someone else 
visits code written yesterday or ten years ago that having code written in a ciear, consistent style 
becomes extremely useful. Understandable code frees mental bandwidth from having to puzzle out 
inconsistencies, making it easier to maintain and enhance projects of aU sizes. 

What this means is that you should go the extra mile to make your code as readable as possible: 

>- Avoid abbreviating variable names. 

>- Write out your function argument names. 

>- Document your classes and methods. 

>- Comment your code. 

>- Refactor repeated lines of code into reusable functions or methods. 

>- Keep functions and methods short. A good rule of thumb is that scroUing should not be nec- 
essary to read an entire function or method. 

When you come back to your code after time away from it, you’ll have an easier time picking up 
where you left off. 

Take those pesky abbreviated variable names, for example. When you see a variable called 
balance_sheet_decrease , it’s much easier to interpret in your mind than an abbreviated variable 
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like bsd or bal s d . These types of shortcuts may save a few seconds of typing, but those savings 
comes at the expense of hours or days of technical debt. It’s not worth it. 

1.2 PEP 8 

PEP8 is the official style guide for Python. We advise reading it in detail and learn to foUow the 
PEP 8 coding conventions: python. org/dev/peps/pep-0008/ 

PEP 8 describes coding conventions such as: 

>- “Use 4 spaces per indentation level.” 

>■ “Separate top-level function and class definitions with two blank lines.” 

>- “Method definitions inside a class are separated by a single blank line.” 

All the Python files in your Django projects should follow PEP 8. If you have trouble remembering 
the PEP 8 guidelines, find a plugin for your code editor that checks your code as you type. 

When an experienced Python programmer sees gross violations of PEP 8 in a Django project, even 
if they don’t say something mean, they are probably thinking bad things. Trust us on this one. 


WARNING: Don’t Change an Ejdsting Project’s Conventions 


The style of PEP 8 applies to new Django projects only. If you are brought into an exist- 
ing Django project that foUows a different convention than PEP 8, then foUow the existing 
conventions. 

Please read the “A Foolish Consistency is the Hobgoblin of Little Minds” section of PEP 8 
for details about this and other reasons to break the rules: 

>- python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the- 
hobgoblin-of-little-minds 


PACKAGE TIP: Use FlakeS for Checking Code Quality 


Created by Tarek Ziade and now maintained by the PyCQA group, this is a very useful 
command-line tool for checking coding style, quality, and logic errors in projects. Use while 
developing locally and as a component of Continuous Integration. 
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1.3: The Word on Imports 


1.2.1 The 79-Character Limit 


Nojoke, I stili deal with consoles that are restricted to 80 characters. 

- Barry Morrison, Systems Engineer and tech reviewer of Two Scoops of Django. 

According to PEP 8, the limit of text per line is 79 characters. This exists because it’s a safe value that 

most text-wrapping editors and developer teams can accommodate without hurting the understand- 

ability of code. 

However, PEP 8 also has a provision for relaxing this limit to 99 characters for exclusive team projects. 

We interpret this to mean projects that are not open source. 

Our preference is as follows: 

>- On open source projects, there should be a hard 79-character limit. Our experience has shown 
that contributors or visitors to these projects will grumble about line length issues; however, it 
has not kept contributors away and we feel the value isn’t lost. 

>■ On private projects, we relax the limit to 99 characters, taking fuU advantage of modern mon- 
itors. 

Please read python.org/dev/peps/pep-0008/#maximum-line-length 


TIP: Aymeric Augustin on Line Length Issues 


Django core developer Aymeric Augustin says, “Fitting the code in 79 columns is never a 
good reason to pickworse names for variables, functions, and classes. It’s much more impor¬ 
tant to have readable variable names than to fit in an arbitrary limit of hardware from three 
decades ago.” 


1.3 The Word on Imports 

PEP 8 suggests that imports should be grouped in the following order: 

O Standard library imports 
0 Related third-party imports 
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© Local applicatiori or library specific imports 
When we’re working on a Django project, our imports look something like the following: 



(Note: you dont actually need to comment your imports like this. The comments are just here to 
explain the example.) 

The import order in a Django project is: 

O Standard library imports. 

0 Imports from core Django. 

© Imports from third-party apps including those unrelated to Django. 

O Imports from the apps that you created as part of your Django project. (YouU read more about 
apps in Chapter 4: Fundamentals of Django App Design.) 


1.4 Use Explicit Relative Imports 

When writing code, it’s important to do so in such a way that it’s easier to move, rename, and version 
your Work. In Python, explicit relative imports remove the need for hardcoding a module’s package via 
implicit relative imports, separating individual modules from being tightly coupled to the architecture 
around them. Since Django apps are simply Python packages, the same rules apply. 
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1.4: Use Explicit Relative Imports 


To illustrate the benefits of explicit relative imports, let’s explore an example. 

Imagine that the following snippet is from a Django project that you created to track your ice cream 
consumption, including all of the waffle/sugar/cake cones that you have ever eaten. 

Oh no, your cones app contains implicit relative imports, which are bad! 


Example 1.2: Bad Python Imports 

# cones/views.py 

from django.views.generic import CreateView 

# DON'T DO THIS! 

# Hardcoding of the 'cones' package 

# with implicit relative imports 
from cones.models import WaffleCone 
from cones.forms import WaffleConeForm 
from core.views import FoodMixin 

class WaffleConeCreateView( FoodMixin, CreateView) 
model = WaffleCone 
form_class = WaffleConeForm 


Sure, your cones app works fine within your ice cream tracker project, but it has those nasty implicit 
relative imports that make it less portable and reusable: 

>- What if you wanted to reuse your cones app in another project that tracks your general dessert 
consumption, but you had to change the name due to a naming conflict (e.g. a conflict with a 
Django app for snow cones)? 

>- What if you simply wanted to change the name of the app at some point? 

With implicit relative imports, you can’t just change the name of the app; you have to dig through 
all of the imports and change them as well. It’s not hard to change them manually, but before you 
dismiss the need for explicit relative imports, keep in mind that the above example is extremely simple 
compared to a real app with various additional utility modules. 

Let’s now convert the bad code snippet containing implicit relative imports into a good one contain- 
ing explicit relative imports. Heres the corrected example: 
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Another concrete advantage is that we can immediately teli our local/internal imports from global/ex- 
ternal imports, highlighting the Python package as a unit of code. 


To summarize, here’s a table of the different Python import types and when to use them in Django 
projects: 


Code 

Import Type 

Usage 

from core.views import FoodMixin 

absolute import 

Use when importing from outside the 
current app 

from .models import WaffleCone 

explicit relative 

Use when importing from another 
module in the current app 

from models import WaffleCone 

implicit relative 

Often used when importing from 
another module in the current app, but 
not a good idea 


Table 1.1: Imports: Absolute vs. Explicit Relative vs. Implicit Relative 


Get into the habit of using explicit relative imports. It’s very easy to do, and using explicit relative 
imports is a good habit for any Python programmer to develop. 
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1.5: Avoid Usingimport 


TIP: Doesn’t PEP 328 Clash With PEP 8? 


See what Guido van Rossum, BDFL of Python says about it: 

>- python.org/pipermai1/python-dev/2010-October/104476.html 


Additional reading: python. org/dev/peps/pep-0328/ 

1.5 Avoid Using Import * 

In 99 percent of ali our work, we explicitly import each module: 



Never do the foUowing: 


Example 1.5: Import * 

# ANTI-PATTERN: Don't do this! 
from django.forms import * 
from django.db.models import * 


The reason for this is to avoid implicitly loading aU of another Python module’s locals into and over 
our current modules namespace, this can produce unpredictable and sometimes catastrophic results. 

We do cover a specific exception to this rule in Chapter 5: Settings and Requirements Files. 

Let’s look at the bad code example above. Both the Django forms and Django models libraries have a 
class called CharField . By implicitly loading both libraries, the models library overwrote the forms 
version of the class. This can also happen with Python built-in libraries and other third-party libraries 
overwriting critical functionality. 

Using import * is like being that greedy customer at an ice cream shop who asks for a free taster 
spoon of aU thirty-one flavors, but who only purchases one or two scoops. Don’t import everything 
ifyoure only going to use one or two things. 
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If the customer then walked out with a giant ice cream bowl containing a scoop of every or almost 
every flavor, though, it would be a different matter. 



Figure 1.1: Using import * in an ice cream shop. 


1.5.1 Other Python Naming Collisions 


You 11 run into similar problems if you try to import two things with the same name, such as: 


Example 1.6: Python Module CoUisions 

# ANTI-PATTERN: Don't do this! 

from django.db.models import CharField 

from django.forms import CharField 


If you need to avoid a naming collision of this nature, you can always use aliases to overcome them: 



1.6 Django Coding Style 

This section covers both the official guidelines as well as unofficial but commonly-accepted Django 
conventions. 
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1.6: Django Coding Style 


1.6.1 Consider the Django Coding Style Guidelines 

It goes without saying that it’s a good idea to be aware of common Django style conventions. In fact, 
internally Django has its own set of style guidelines that extend PEP 8: 

>- docs.djangoproj ect.com/en/1.11/internals/contributing/writing-code/ 
coding-style/ 

Additionally, while the foUowing are not specified in the official standards, they are common enough 
in the Django community that you wiU probably want to follow them in your projects. 


TIP: Review the Documentation on Django Internals 


The documentation on Django internals hold a lot more than just coding style. Theyre chock- 
full of useful Information, including the history of the Django project, the release process, and 
more! We recommend you check them out. 

docs.dj angoproj ect.com/en/1.11/internals/ 


1.6.2 Use Underscores in URL Pattern Names Rather Than Dashes 

We always try to use underscores (the character) over dashes. This isn’t just more Pythonic, it’s 
friendlier to more IDEs and text editors. Note that we are referring to the name argument of url() 
here, not the actual URL typed into the browser. 

The wrong way, with dashes in uri names: 


Example 1.8: Bad URL Pattern Names 

patterns = [ 

url(regex= ' ''add/$ ' , 

view=views.add_topping, 
nanie= ' add-toppi ng') , 

] 


The right way, with underscores in uri names: 
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Dashes in actual URLs are fine (e.g. regex=’^add-topping/$’ ). 


1.6.3 Use Underscores in Template Block Names Rather Than Dashes 

For the same reasons as using underscores in URL pattern names, we recommend using underscores 
when defining names of template blocks: in this case theyre more Pythonic and more editor-friendly. 

1.7 Choose JS, HTML, and CSS Style Guides 

1.7.1 JavaScript Style Guides 


Unlike Python which has one oflicial style guide, there is no officia! JavaScript style guide. Instead, 
a numher of unofficial JS style guides have heen created hy various individuals and/or companies: 

https://github.com/feross/standard 

>- Standard combinedJavaScript and Node.js Style Guide github.com/feross/standard 
>- idiomatic.js: Principies of Writing Consistent, Idiomatic JavaScript github.com/ 
rwaldron/idi ornatic.js 

>• Airbnb JavaScript Style Guide github.com/ai rbnb/ j avascri pt 

There is no consensus in the Django or JavaScript communities on any one of these, so just pickyour 
favorite and stick with it. 

However, if you are using a JavaScript framework with a style guide of its own, you should use that 
guide. For example, ember.js has its own style guide. 
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1.8: Never Code to the IDE (Or Text Editor) 


PACKAGE TIP: ESLint: A Pluggable Linting Utility for JavaScript and 
JSX 


ESLint (esli nt. org) is a tool for checking JavaScript and JSX code styles. It has presets 
for the JS style rules of several style guides, including a few of those listed above. There are 
also ESLint plugins for various text editors and ESLint tasks for various JavaScript tools like 
Webpack, Gulp, and Grunt. 


1.7.2 HTML and CSS Style Guides 

>- Code Guide by @mdo for HTML and CSS: codegui de. co 
>• idomatic-css: Principies of Writing Consistent, Idiomatic CSS: 
github.com/necolas/idiomatic-css 


PACKAGE TIP: stylelint 


Stylelint (styleli nt. io) is a coding style formatter for CSS. It checks for consistency 
against the rules for which you configure it for, and it checks the sort order of your CSS 
properties. Just as for ESLint, there are stylelint text editor and task/build tool plugins. 


1.8 Never Code to the IDE (Or Text Editor) 

There are developers who make decisions about the layout and implementation of their project based 
on the features of IDEs (Integrated Development Environment). This can make discovery of project 
code extremely difficult for anyone whose choice of development tool doesnt match the original 
author. 

Always assume that the developers around you like to use their own tools and that your code and 
project layout should be transparent enough that someone stuck using Notepad or Nano will be able 
to navigate your work. 

Eor example, introspecting template tags or discovering their source can be difficult and time 
consuming for developers not using a very, very limited pool of IDEs. Therefore, we follow the 
commonly-used naming pattern of <app_name>_tags. py. 
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1.9 Summary 

This chapter covered our preferred coding style and explained why we prefer each technique. 


Even if you don’t follow the coding style that we use, please follow a consistent coding style. Projects 
with varying styles are much harder to maintain, slowing development and increasing the chances of 
developer mistakes. 
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The Optimal Django Environment 
Setup 


This chapter describes what we consider the best local environment setup for intermediate and ad- 
vanced developers working with Django. 


2.1 Use the Same DataBase Engine Everywhere 

A common developer pitfall is using SQLiteS for local development and PostgreSQL (or MySQL) 
in production. This section applies not only to the SQLiteS/PostgreSQL scenario, but to any scenario 
where youre using two different databases and expecting them to behave identicaUy. 

Here are some of the issues weVe encountered with using different database engines for development 
and production: 


2.1.1 You Can’t Examine an Exact Copy of Production Data Locally 

When your production database is different from your local development database, you can’t grab an 
exact copy of your production database to examine data locally. 

Sure, you can generate a SQL dump from production and import it into your local database, but that 
doesnt mean that you have an exact copy after the export and import. 
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2.1.2 Different Databases Have Different Field Types/Constraints 

Keep in mind that different databases handle typing of field data differently. Djangos ORM attempts 
to accommodate those differences, but theres only so much that it can do. 

For example, some people use SQLiteS for local development and PostgreSQL in production, think- 
ing that the Django ORM gives them the excuse not to think about the differences. Eventually they 
run into problems, since SQLiteS has dynamic, weak typing instead of strong typing. 

Yes, the Django ORM has features that allow your code to interact with SQLiteS in a more strongly 
typed manner, but form and model validation mistakes in development wiU go uncaught (even in 
tests) until the code goes to a production server. You may be saving long strings locaUy without a 
hitch, for example, since SQLiteS won’t care. But then in production, your PostgreSQL or MySQL 
database wiU throw constraint errors that youVe never seen locaUy, and you’ll have a hard time repli- 
cating the issues until you set up an identical database locaUy. 

Most problems usually can’t be discovered untU the project is run on a strongly typed database (e.g. 
PostgreSQL or MySQL). When these types of bugs hit, you end up kicking yourself and scrambling 
to set up your local development machine with the right database. 


TIP: Django+PostgreSQL Rocks 


Most Django developers that we know prefer to use PostgreSQL for aU environments: de¬ 
velopment, staging, QA, and production systems. 

Depending on your operating system, use these instructions: 

>- Mac: Download the one-clickMac installer at postgresapp. com 
>- Windows: Download the one-click Windows installer at 
postgresql.org/download/windows/ 

>- Linux: Install via your package manager, or follow the instructions at 
postgresql.org/download/linux/ 

PostgreSQL may take some work to get running locaUy on some operating systems, but we 
find that it’s weU worth the effort. 
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2.1.3 Fixtures Are Not a Magic Solution 


You may be wondering why you eant simply use fixtures to abstract away the differences between 
your local and production databases. 

Well, fixtures are great for creating simple hardeoded test data sets. Sometimes you need to pre- 
populate your databases with fake test data during development, particularly during the early stages 
of a project. 

Fixtures are not a reliable tool for migrating large data sets from one database to another in a database- 
agnostic way. They are simply not meant to be used that way. Dont mistake the ability of fixtures 
to create basic data (dumpdata/loaddata) with the capability to migrate production data between 
database tools. 


WARNING: Don’t Use SQLite3 with Django in Production 


For any web project with more than one user, or requiring anything but light concurrency, 
SQLiteS is a nightmare in the making. In the simplest terms possible, SQLiteS works great 
in production untrl it doesnt. WeVe experienced it ourselves, and heard horror stories from 
others. 

This issue compounds itselfwith the difiiculty and complexity involved in migrating data out 
of SQLiteS and into something designed for concurrency (e.g., PostgreSQL) when problems 
eventually arise. 

While we’re aware that there are plenty of articles advocating the use of SQLiteS in produc¬ 
tion, the fact that a tiny group of SQLiteS power users can get away with it for particular 
edge cases is not justification for using it in production Django. 
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2.2 Use Pip and Virtualenv 

If you are not doing so already, we strongly urge you to familiarize yourself with both pip and vir- 
tualenv. They are the de facto Standard for Django projects, and most companies that use Django 
rely on these tools. 

Pip is a tool that fetches Python packages from the Python Package Index and its mirrors. It is used 
to manage and instaU Python packages. It’s like easy_install but has more features, the key feature 
being support for virtualenv. 

Virtualenv is a tool for creating isolated Python environments for maintaining package dependencies. 
It’s great for situations where youre working on more than one project at a time, and where there are 
clashes between the version numbers of different libraries that your projects use. 

For example, imagine that youre working on one project that requires Django 1.10 and another that 
requires Django 1.11. 

>• Without virtualenv (or an alternative tool to manage dependencies), you have to reinstall 
Django every time you switch projects. 

>- If that sounds tedious, keep in mind that most real Django projects have at least a dozen 
dependencies to maintain. 

Pip is already included in Python 3.4 and higher. Further reading and installation instructions can 
be found at: 

>- pip: pip. pypa. io 

>■ virtualenv: vi rtualenv. pypa . io 

2.2.1 virtualenvwrapper 

We also highly recommend virtualenvwrapper for Mac OS X and Linux or 
vi rtualenvwrappei—wi n for Windows. The project was started by Doug HeUman. 

Personally, we think virtualenv without virtualenvwrapper can be a pain to use, because every time 
you want to activate a Virtual environment, you have to type something long like: 
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2.2: Use Pip and Virtualenv 



With virtualenvwrapper, you’d only have to type: 



Virtualenvwrapper is a popular companion tool to pip and virtualenv and makes our lives easier, but 
it’s not an absolute necessity. 



Figure 2.1: Pip, virtualenv, and virtualenvwrapper in ice cream bar form. 
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2.3 Install Django and Other Dependencies via Pip 

The official Django documentation describes several ways of installing Django. Our recommended 
installation method is with pip and requirements files. 

To summarize how this worlcs: a requirements file is like a grocery list of Python packages that you 
want to install. It contains the name and optionally suitable version range of each package. You use 
pip to install packages from this list into your Virtual environment. 

We cover the setup of and installation from requirements files in Chapter 5: Settings and Require¬ 
ments Files. 


TIP: SettingPYTHONPATH 


If you have a firm grasp of the command line and environment variables, you can set your 
virtualenv PYTHON PATH so that the django-admin.py command can be used to serve your site 
and perform other tasks. 

You can also set your virtualenvs PYTHON PATH to include the current directory with the latest 
version of pip. Running “pi p i nstall -e . ” from your project’s root directory will do the 
trick, installing the current directory as a package that can be edited in place. 

If you don’t know how to set this or it seems complicated, dont worry about it and stick 
with manage.py. 

Additional reading: 

>- hope.si mons- rock.edu/~pshi elds/cs/python/pythonpath.html 
>- docs.djangoproject.com/en/1.11/ref/django-admin/ 


2.4 Use a Version Control System 

Version control Systems are also known as revision control or source control. Whenever you work on 
any Django project, you should use a version control system to keep track of your code changes. 

Wikipedia has a detailed comparison of different version control systems: 

>- en.wikipedi a.org/wiki/Compa rison_of_revision_control_softwa re 
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2.S: Optional: Identical Environments 


Of ali the options, Git is the most popular among Django, Python, and JavaScript developers. Git 
makes it easy to create branches and merge changes. 

When using a version control System, it’s important to not only have a local copy of your code reposi- 
tory, but also to use a code hosting Service for backups. For this, we recommend that you use GitHub 
(github. com) or GitLab (gitlab. com). 

2.5 Optional: Identical Environments 

What Works on a programmers laptop might not work in production. But what if your local devel- 
opment environment was identical to your project’s staging, test, and production environments? 

Of course, if the production infrastructure consists of 10,000 servers, it’s completely unrealistic to 
have another 10,000 local servers for development purposes. So when we say identical, we mean “as 
identical as realisticaUy possible.” 

These are the environment differences that you can eliminate: 

Operating system differences. If we’re developing on a Mac or on Windows, and if our site is de- 
ployed on Ubuntu Linux, then there are huge differences between how our Django project 
Works locally and how it works in production. 

P3rthon Setup differences. Let’s face it, many developers and sysadmins don’t even know which ver¬ 
sion of Python they have running locally, although no one will admit it. Why? Because setting 
up Python properly and understanding your setup completely is hard. 
Developer-to-developer differences. On large development teams, a lot of time can be wasted try- 
ing to debug differences between one developers setup and anothers. 

The most common way to set up identical development environments is with Docker. 


2.5.1 Docker 

At the time of this writing, Docker is the industry Standard for containerization of environments. 
It has excellent support across aU operating systems, including Microsoft Windows. Working with 
Docker is sort of like developing inside of a VM, except more lightweight. Docker containers share 
the host OS but have their own isolated process and memory space. Furthermore, since Docker uses 
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a uni on-capablefi lesystem, containers can be built quickly off of a snapshot plus deitas rather 
than building from scratch. 

For the purposes of local development, its main benefit is that it makes setting up environments that 
closely match development and production much easier. 

For example, if our development laptops run OS X (or Windows, or Centos, etc) but a project’s 
configuration is Ubuntu-specific, we can use Docker via Docker Compose to quickly get a Virtual 
Ubuntu development environment set up locally, complete with aU the packages and setup configu- 
rations needed for the project. We can: 

>- Set up identical local development environments for everyone on our project’s dev team. 

>- Configure these local development environments in a way similar to our staging, test, and 
production servers. 

The potential downsides are: 

>- Extra complexity that is not needed in many situations. For simpler projects where we’re not 
too worried about OS-level differences, it’s easier to skip this. 

>■ On older development machines, running even lightweight containers can slow performance 
to a crawl. Even on newer machines, smaU but noticeable overhead is added. 

References for developing with Docker: 

>- cookiecutter-dj ango.readthedocs.io/en/latest/ 
developing-locally-docker.html 

>- http: //bi t. ly/ IdWnzVW Real Python article on Django and Docker Compose 
>- dockerbook.com 

2.6 Summary 

This chapter covered using the same database in development as in production, pip, virtualenv, and 
version control Systems. These are good to have in your tool chest, since they are commonly used not 
just in Django, but in the majority of Python Software development. 
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How to Lay Out Django Projects 


Project layout is one of those areas where core Django developers have differing opinions about 
what they consider best practice. In this chapter, we present our approach, which is one of the most 
commonly-used ones. 


PACKAGE TIP: Django Project Templates 


There are a number of project templates that reaUy kickstart a Django project and foUow the 
patterns described in this chapter. Here are two links that may be of use when we bootstrap 
a project: 

>- github.com/pydanny/cookiecutter-django 
Featured in this chapter. 

>- djangopackages.org/grids/g/cookiecutters/ 

A list of alternate cookiecutter templates. 


3.1 Django 1.1 l’s Default Project Layout 

Let’s examine the default project layout that gets created when you run startproject and startapp: 



Here’s the resulting project layout: 
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Example 3.2: Default Project Layout 


mysite/ 
h- manage.py 

I - niy_app 

I I - __-init__.py 

I I- admin.py 

I I — apps.py 

I h- migrations 

I I I—_init_ 

I I— models.py 

I I- tests.py 

I I- views.py 

I- mysite 

I__init_. py 

K settings.py 

I- urls.py 

I- wsgi.py 


There are a number of problems with Djangos default project layout. While useful for the tutorial, 
it’s not quite as useful once you are trying to put together a real project. The rest of this chapter will 
explain why. 


3.2 Our Preferred Project Layout 

We rely on a modified two-tier approach that builds on what is generated by the 
dj ango-admi n. py startproj ect management command. Our layouts at the highest level are: 


Example 3.3: Project Root Levels 


< repository_root>/ 
h- <configuration_root>/ 

I- <django_project_root>/ 


Let’s go over each level in detail: 
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3.2: Our Preferred Project Layout 


3.2.1 Top Level: Repository Root 


The <repository_root> directory is the absolute root directory of the project. In addition to the 
<django^roject_root> and <configuration_root>, we also include other critical components like the 
README.rst, docs/ directory, manage.py, .gitignore, requirements.txt files, and other high-level files 
that are required for deployment and running the project. 







Figure 3.1: Yet another reason why repositories are important. 


TIP: Common Practice Varies Here 


Some developers like to combine the <django_project_root> into the <repository_root> of the 
project. 


3.2.2 Second Level: Django Project Root 

The <django^roject_root>/ directory is the root of the actual Django project. Non-configuration 
Python code files are inside this directory, its subdirectories, or below. 

If using d j ango-admi n. py startproj ect, you would run the command from within the repos¬ 
itory root. The Django project that it generates would then be the project root. 

3.2.3 Second Level: Configuration Root 

The <configuration_root> directory is where the settings module and base URLConf {urls.py) are 
placed. This must be a validPythonpackage (containing an _init_.py module). 
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If using d j ango-admi n . py sta rtpro j ect, the configuration root is initially inside of the Django 
project root. It should be moved to the repository root. 

The files in the configuration root are part of what is generated by the 
dj ango-admi n. py startproj ect command. 


Figure 3.2: Three-tiered scoop layout. 



3.3 Sample Project Layout 

Let’s take a common example: a simple rating site. Imagine that we are creating Ice Cream Ratings, 
a web application for rating different brands and ffavors of ice cream. 

This is how we would lay out such a project: 


Example 3.4: Layout for icecreamratings 


icecreamratings_proj ect 
h- config/ 

I I- settings/ 



I- uris.py 
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3.3: Sample Project Layout 


h- 

h- 


h- 

h- 

h- 

h- 


' - wsgi.py 

docs/ 

icecreamratings/ 

I— media/ # Development only! 
I— Products/ 

I— profiles/ 

I— ratings/ 

I— static/ 

I- templates/ 

■gitignore 
Makefi le 
README.rst 
manage.py 
requirements.txt 


Let’s do an in-depth review of this layout. As you can see, in the icecreamratingsproject/ directory, 
which is the <repository_root> , we have the following files and directories. We describe them in the 
table below: 


File or Directory 

Purpose 

■gitignore 

Lists the files and directories that Git should ignore. (This file is 
different for other version control Systems. For example, if you are 
using Mercurial instead, youd have an .hgignore file.) 

config/ 

The <conJiguration_root> of the project, where project-wide settings, 
urh.py, and wsgi.py modules are placed (WeTl cover settings layout 
later in Chapter 5: Settings and Requirements Files). 

Makefile 

Contains simple deployment tasks and macros. For more complex 
deployments you may want to rely on tools like Invoke, Paver, or 

Fabri c. 

manage.py 

If you leave this in, don’t modify its contents. Refer to Chapter 5: 
Settings and Requirements Files for more details. 

README.rst and docs/ 

Developer-facing project documentation. We cover this in 

Chapter 23: Documentation: Be Obsessed. 
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FileorDirectory Purpose 


A list of Python packages required by your project, including the 
requirements.txt Django 1.11 package. You’U read more about this in Chapter 21: 

Djangos Secret Sauce: Third-Party Packages. 

icecreamratings/ The <django_project_root> of the project. 

Table 3.1: Repository Root Files and Directories 


When anyone visits this project, they are provided with a high-level view of the project. WeVe found 
that this aUows us to work easily with other developers and even non-developers. For example, it’s 
not uncommon for designer-focused directories to be created in the root directory. 


Inside the icecreamratings^roject/icecreamratings directory, at the <django^roject_root>, we place 
the foUowing files/directories: 


File or Directory 

Purpose 


media/ 

For use in development only: user-generated static media assets such as 
photos uploaded by users. For larger projects, this will be hosted on separate 
static media server(s). 

producis/ 

App for managing and displaying ice cream brands. 

profiles/ 

App for managing and displaying user profiles. 

ratings/ 

App for managing user ratings. 

static/ 

Non-user-generated static media assets including CSS, JavaScript, and 
images. For larger projects, this will be hosted on separate static media 
server(s). 

templates/ 

Where you put your site-wide Django templates. 


Table 3.2: Django Project Files and Directories 
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3.4: WhatAboutthe Virtualenv? 


TIP: Conventions for Static Media Directory Names 


In the example above, we foUow the official Django documentations convention of using 
static/ for the (non-user-generated) static media directory. 

If you find this confusing, theres no harm in caUing it assets/ or site_assets/ instead. Just 
remember to update your STATICFILES_DIRS setting appropriately. 


3.4 What About the Virtualenv? 

Notice how there is no virtualenv directory anywhere in the project directory or its subdirectories? 
That is completely intentional. 

A good place to create the virtualenv for this project would be a separate directory where you keep 
ali of your virtualenvs for aU of your Python projects. We like to put all our environments in one 
directory and aU our projects in another. 



Figure 3.3: An isolated environment, allowing your ice cream to swim freely. 


For example, on Mac OS X or Linux: 
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On Windows: 


Example 3.6: On Windows 


c: \projects\i cecreamratings_proj ect\ 
c:\envs\"icecreamrat"ings\ 


If youre using virtualenvwrapper (Mac OS X or Linux) or virtualenvwrapper-win (Windows), that 
directory defaults to -/.virtualenvs/ and the virtualenv would be located at: 


Example 3.7: virtualenvwrapper 


~/.virtualenvs/icecreamratings/ 


Also, remember, there’s no need to keep the contents of your virtualenv in version control since it 
already has all the dependencies captured in requirements.txt, and since you won’t be editing any of 
the source code files in your virtualenv directly. Just remember that requirements.txt does need to 
remain in version control! 


3.4,1 Listing Current Dependencies 

If you have trouble determining which versions of dependencies you are using in your virtualenv, at 
the command line you can list your dependencies by typing: 


Example 3.8: Listing Current Dependencies 


$ pip freeze 


With Mac or Linux, you can pipe this into a requirements.txt file: 


Example 3.9: Saving Current Dependencies to a File 


$ pip freeze > requirements.txt 
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3.5 Going Beyond startproject 

Djangos startproject command allows you to create and use simple Django project templates. 
However, over time the Controls (deployment, front end tooling, etc) around a project grow more and 
more complex. Most of us hit the limitations of startproj ect quickly and need a more powerful 
project templating tool. Hence the use of Cookiecutter, an advanced project templating tool that 
can be used for generating Django project boilerplate. 


TIP: Audrey on Cookiecutter 


I originally created Cookiecutter in 2013 to meet my own Python package boilerplate creation 
needs. It was the first project to template file paths and file contents identicaUy, an idea I 
thought was siUy but decided to implement anyway. 

There are now Cookiecutter templates for Python, C, C++, Common Lisp, JS, LaTeX/Xe- 
TeX, Berkshelf-Vagrant, HTML, Scala, 6502 Assembly, and more. 

Cookiecutter isn’t just a command-line tool, it’s a library used by a host of organizations. You 
can also find it integrated into IDEs such as PyCharm and Visual Studio. 


In this section, we present our version of the ultimate Django project template, rendered by Cook¬ 
iecutter. 


3.5.1 Generating Project Boilerplate With Cookiecutter 

Here’s how Cookiecutter works: 

O First, it asks you to enter a series of values (e.g. the value for proj ect_name). 

0 Then it generates aU your boilerplate project files based on the values you entered. 

On Python 2.7+ or 3.6+, youTl first need to instaU Cookiecutter as per the instructions in the official 
Cookiecutter documentation. 
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3.5.2 Generating a Starting Project With Cookiecutter Django 

Here’s how you would use Cookiecutter to generate your Django 1.11 boilerplate from Cooki ecut- 
ter Django: 



After fiUing out ali the values, in the directory where you ran Cookiecutter, it wiU create a direc- 
tory for your project. In this case with the values entered above, the name of this directory wiU be 
icecreamratingsproject. 

The resulting project files wiU be roughly similar to the layout example we provided. The project will 
include settings, requirements, starter documentation, a starter test suite, and more. 
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TIP: What Are All the Other Files? 


Keep in mind that Cookiecutter Django goes much further than the basic project layout 
components that we outlined earlier in this chapter. It’s our ultimate Django project template 
that we use for our projects, so it has a lot of other bells and whistles. 

It’s a lot fancier than the default startproj ect template provided by Django. Wed rather 
have you see our actual, real-life template that we use for our projects than a stripped-down, 
beginner-oriented template that we dont use. 


You are welcome to fork Cooki ecutter Dj ango and customize it to fit your own Django project 
needs. 


3.5.3 Other Alternatives 

People can get very opinionated about their project layout being the “right” way, but as we mentioned, 
theres no one right way. 

It’s okay if a project differs from our layout, just so long as things are either done in a hierarchical fash- 
ion or the locations of elements of the project (docs, templates, apps, settings, etc) are documented 
in the root README.rst. 


We encourage you to explore the forks of Cookiecutter Django, and to search for other Cookiecutter- 
powered Django project templates online. You’ll learn all kinds of interesting tricks by studying other 
people’s project templates. 



Figure 3.4: Project layout differences of opinion can cause ice cream fights. 
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3.6 Summary 

In this chapter, we covered our approach to basic Django project layout. We provided a detailed 
example to give you as much insight as possible into our practices. 

Project layout is one of those areas of Django where practices differ widely from developer to devel- 
oper and group to group. What works for a smaU team may not work for a large team with distributed 
resources. Whatever layout is chosen should be documented clearly. 
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4 


Fundamentals of Django App 
Design 


It’s not uncommon for new Django developers to become understandably confused by Django’s usage 
of the Word “app”. So before we get into Django app design, it’s very important that we go over some 
definitions. 


A Django project is a web application powered by the Django web framework. 

Django apps are small libraries designed to represent a single aspect of a project. A Django project 
is made up of many Django apps. Some of those apps are internal to the project and will never 
be reused; others are third-party Django packages. 

INSTALLED_APPS is the list of Django apps used by a given project available in its IN- 
STALLED_APPS setting. 

Hiird-party Django packages are simply pluggable, reusable Django apps that have been packaged 
with the Python packaging tools. We’llbegin coverage of them in Chapter 21: Djangos Secret 
Sauce: Third-Party Packages. 



Yoc'!?, Pj^VC.0 

Pftejecr 1^ 4 


Figure 4.1: It’ll make more sense when you see the next figure. 
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Figure 4.2: Did that make sense? If not, read it again. 
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4.1 The Golden Rule of Django App Design 


James Bennett is a Django core developer. He taught us everything that we know about good Django 
app design. We quote him: 


“The art of creating and maintaining a good Django app is that it should follow the 
truncated Unix philosophy according to Douglas Mcllroy: ‘Write programs that do one 
thing and do it well.’” 


In essence, each app should be tightly focused on its task. If an app caht be explained in a single 
sentence of moderate length, or you need to say ‘and’ more than once, it probably means the app is 
too big and should be broken up. 
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4,1.1 A Practical Example of Apps in a Project 


Imagine that we’re creating a web application for our fictional ice cream shop called “Two Scoops”. 
Picture us getting ready to open the shop: polishing the countertops, making the first batches of ice 
cream, and building the website for our shop. 


We’d call the Django project for our shop’s website twoscoopsproject. The apps within our Django 
project might be something like: 

>- Kflavors app to track all of our ice cream flavors and list them on our website. 

>- A blog app for the official Two Scoops blog. 

>- An events app to display listings of our shop’s events on our website: events such as Strawberry 
Sundae Sundays and Fudgy First Fridays. 

Each one of these apps does one particular thing. Yes, the apps relate to each other, and you could 
imagine events or blog posts that are centered around certain ice cream flavors, but it’s much better 
to have three specialized apps than one app that does everything. 


In the future, we might extend the site with apps like: 

>- A shop app to allow us to seU pints by mail order. 

>- A tickets app, which would handle ticket sales for premium all-you-can-eat ice cream fests. 

Notice how events are kept separate from ticket sales. Rather than expanding the events app to sell 
tickets, we create a separate tickets app because most events dont require tickets, and because event 
calendars and ticket sales have the potential to contain complex logic as the site grows. 


EventuaUy, we hope to use the tickets app to seU tickets to Icecreamlandia, the ice cream theme park 
fllled with thriU rides that weVe always wanted to open. 


Did we say that this was a flctional example? Ahem...well, heres an early concept map of what we 
envision for Icecreamlandia: 
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Figure 4.3: Our vision for Icecreamlandia. 


4.2 What to Name Your Django Apps 

Everyone has their own conventions, and some people like to use really colorful names. We like to 
use naming systems that are dull, boring, and obvious. In fact, we advocate doing the following: 

When possible keep to single word names liks Jiavors, animals, blog, polis, dreams, estimates, and 
finances. A good, obvious app name makes the project easier to maintain. 

As a general rule, the apps name should be a plural version of the app’s main model, but there are 
many good exceptions to this rule, blog being one of the most common ones. 

Dont just consider the apps main model, though. You should also consider how you want 
your URLs to appear when choosing a name. If you want your sites blog to appear at 
http://www.example.com/weblog/, then consider naming your app weblog rather than blog, posts, 
or blogposts, even if the main model is Post, to make it easier for you to see which app corresponds 
with which part of the site. 

Use valid, PEP 8-compliant, importable Python package names: short, all-lowercase names with- 
out numbers, dashes, periods, spaces, or special characters. If needed for readability, you can use 
underscores to separate words, although the use of underscores is discouraged. 
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4.3 When in Doubt, Keep Apps Small 

Dont worry too hard about getting app design perfect. It’s an art, not a Science. Sometimes you have 
to rewrite them or break them up. That’s okay. 


Try and keep your apps small. Remember, it’s better to have many small apps than to have a few giant 
apps. 


Good 



Figure 4.4: Two small, single-flavor pints are better than a giant, 100-flavor Container. 


4.4 What Modules Belong in an App? 

In this section we cover both the common and uncommon Python modules that belong in an app. 
For those with even a modicum of experience with Django, skipping to Section 4.4.2: Uncommon 
App Modules may be in order. 


4,4,1 Common App Modules 

Here are common modules seen in 99% of Django apps. These will prove very familiar to most readers, 
but we’re placing this here for those just coming into the world of Django. For reference, any module 
ending with a slash (“/’) represents a Python package, which can contain one or more modules. 
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Example 4.1: Common App Modules 


# Common modules 
scoops/ 

_i ni t_. py 

I— admin.py 

I- forms.py 

I- management/ 

I- migrations/ 

I- models.py 

I- templatetags/ 

I- tests/ 

I- uris.py 

h- views.py 

Over time a convention of module names has emerged for building Django apps. By following this 
convention across building of apps we set behaviors for ourselves and others, making examining each 
others code easier. While Python and Django are flexible enough that most of these dont need to be 
named according to this convention, doing so will cause problems. Probably not from an immediate 
technical perspective, but when you or others look at nonstandard module names later, it will prove 
to be a frustrating experience. 

4,4,2 Uncommon App Modules 

Here are less common modules, which may or may not be familiar to many readers: 


Example 4.2: Uncommon Django Modules 


# uncommon modules 
scoops/ 

I— api/ 

I- behaviors.py 

I- constants.py 

I— context_processors.py 
I- decorators.py 

I- db/ 
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V- 

exceptions. 

py 

h- 

fields.py 


h- 

factori es.py 

h- 

helpers.py 


h- 

managers.py 


h- 

middleware. 

py 

h- 

signals.py 


h- 

utiIs.py 


h- 

viewmixins. 

py 


What is the purpose of each module? Most of these should be obviously named, but we’ll go over a 
few that might not be so ciear. 

api/ : This is the package we create for isolating the various modules needed when creating an api. 

See Section 16.3.1: Use Consistent API Module Naming. 
behaviors.py : An option for locating model mbdns per Section 6.7.1: Model Behaviors a.k.a Mbc- 
ins. 

constants.py : A good name for placement ofapp-level settings. Ifthere are enough ofthem involved 
in an app, breaking them out into their own module can add clarity to a project. 
decorators.py Where we like to locate our decorators. For more Information on decorators, see 
Section 9.3: Decorators Are Sweet. 

db/ A package used in many projects for any custom model fields or components. 
fields.py is commonly used for form fields, but is sometimes used for model fields when there isnt 
enough field code to justify creating a db/ package. 
factories.py Where we like to place our test data factories. Described in brief in Section 22.3.5: 
Don’t Rely on Fixtures 

helpers.py What we call helper functions. These are where we put code extracted from views (Sec¬ 
tion 8.5: Try to Keep Business Logic Out of Views) and models (Section 6.7: Understanding 
Fat Models) to make them lighter. Synonymous with utils.py 
managers.py When models.py grows too large, a common remedy is to move any custom model 
managers to this module. 

signals.py While we argue against providing custom signals (see Chapter28: Signals: Use Cases 
and Avoidance Techniques), this can be a useful place to put them. 
utils.py Synonymous with helpers.py 

viewmixins.py View modules and packages can be thinned by moving any view mbdns to this mod¬ 
ule. See Section 10.2: Using Mixins With CBVs. 
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For ali of the modules listed in this section, their focus should be at the ‘app-level’, not global tools. 
Global-level modules are described in Section 29.1: Create a Core App for Your Utilities. 

4.5 Summary 

This chapter covered the art of Django app design. SpecificaUy, each Django app should be tightly- 
focused on its own task, possess a simple, easy-to-remember name. If an app seems too complex, it 
sbould be broken up into smaller apps. Getting app design right takes practice and elfort, but it’s 
well wortb the effort. 
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Settings and Requirements Files 


Django 1.11 has over 150 settings that can be controlled in the settings module, most of which 
come with default values. Settings are loaded when your server starts up, and experienced Django 
developers stay away from trying to change settings in production since they require a server restart. 



Figure 5.1: As your project grows, your Django settings can get pretty complex. 


Some best practices we like to follow: 

>- 7\11 settings files need to be version-controlled. This is especially true in production environ- 
ments, where dates, times, and explanations for settings changes absolutely must be tracked. 

>- Don’t Repeat Yourself. You should inherit from a base settings file rather than cutting-and- 
pasting from one file to another. 

>- Keep secret keys safe. They should be kept out of version control. 
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5.1 Avoid Non-Versioned Local Settings 

We used to advocate the non-versioned local_settings anti-pattern. Now we know better. 

As developers, we have our own necessary settings for development, such as settings for debug tools 
which should be disabled (and often not installed to) staging or production servers. 

Furthermore, there are often good reasons to keep specific settings out of public or private code 
repositories. The SECRET_KEY setting is the first thing that comes to mind, but API key settings to 
Services like Amazon, Stripe, and other password-type variables need to be protected. 


WARNING: Protect Your Secrets! 


The SECRET_KEY setting is used in Djangos cryptographic signing functionality, and needs 
to be set to a unique, unpredictable setting best kept out ofversion control. Running Django 
with a known SECRET_KEY defeats many of Djangos security protections, which can lead 
to serious security vulnerabilities. For more details, read docs.djangoproject.com/en/ 
1.11/topics/signing/. 

The same warning for SECRET_KEY also applies to production database passwords, AWS 
keys, OAuth tokens, or any other sensitive data that your project needs in order to operate. 

Later in this chapter we’U show how to handle the SECRET_KEY issue in the “Keep Secret 
Keys Out With Environment Settings” section. 


A common solution is to create local_settings.py modules that are created locally per server or develop¬ 
ment machine, and are purposefully kept out of version control. Developers now make development- 
specific settings changes, including the incorporation ofbusiness logic without the code being tracked 
in version control. Staging and deployment servers can have location specific settings and logic with¬ 
out them being tracked in version control. 

What could possibly go wrong?!? 

Ahem... 

>- Every machine has untracked code. 

>- ITow much hair will you pull out, when after hours of failing to duplicate a production bug 
locally, you discover that the problem was custom logic in a production-only setting? 
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>■ How fast will you run from everyone when the ‘bug’ you discovered locally, fixed and pushed 
to production was actually caused by customizations you made in your own local_settings.py 
module and is now crashing the site? 

>- Everyone copy/pastes the same local_settings.py module everywhere. Isnt this a violation of 
Don’t Repeat Yourself but on a larger scale? 


Let’s take a different approach. Let’s break up development, staging, test, and production settings 
into separate components that inherit from a common base object in a settings file tracked by version 
control. Plus, we’ll make sure we do it in such a way that server secrets will remain secret. 


Read on and see how it’s done! 


5.2 Using Multiple Settings FUes 


TIP: History of This Setup Pattern 


The Setup described here is based on the so-caUed “The One True Way”, from Jacob Kaplan- 
Moss’ The Best (and Worst) of Django talk at OSCON 2011. See slideshare.net/ 
j acobian/the-best-and-worst-of-dj ango. 


Instead of having one settings.py file, with this setup you have a settings/ directory containing your 
settings files. This directory will typically contain something like the following: 
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WARNING: Requirements + Settings 


Each settings module should have its own corresponding requirements file. We’ll cover this 
at the end of this chapter in Section 5.5: Using Multiple Requirements Files. 


Settings file Purpose 


base.py 

Settings common to all instances of the project. 

local.py 

This is the settings file that you use when youre working on the project locaUy. 
Local development-specific settings include DEBUG mode, loglevel, and 
activation of developer tools like django-dehug-toolhar. 

staging.py 

Staging version for running a semi-private version of the site on a production 
server. This is where managers and clients should he looking hefore your work is 
moved to production. 

test.py 

Settings for running tests including test runners, in-memory datahase 
definitions, and log settings. 

production.py 

This is the settings file used by your live production server(s). That is, the 
server(s) that host the real live website. This file contains production-level 
settings only. It is sometimes prod.py. 

Table 5.1: Settings files and their purpose 


TIP: Multiple Files With Continuous Integration Servers 


You’ll also want to have a ci.py module containing that servers settings. Similarly, ifit’s alarge 
project and you have other special-purpose servers, you might have custom settings files for 
each of them. 


Let’s take a look at how to use the shell and runserver management commands with this setup. 
You’ll have to use the — setti ngs command line option, so you’ll he entering the following at the 
command-line. 

To start the Python interactive interpreter with Django, using your settings/local.py settings file: 
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To run the local development server with your settings/local.py settings file: 



TIP: DJANGO_SETTINGS_MODULE and PYTHONPATH 


A great alternative to using the — settings command line option everywhere is to set 
the D3ANG0_SETTINGS_M0DULE and PYTHONPATH environment variahle to your desired 
settings module path. Youd have to set DTANGO_SETTINGS_MODULE to the corresponding 
settings module for each environment, of course. 

For those with a more comprehensive understanding of virtualenvwrapper, another alterna¬ 
tive is to set DOANGO_SETTINGS_HODULE and PYTHONPATH in the postactivate script 
and unset them in the postdeacti vate script. Then, once the virtualenv is activated, 
you can just type python from anywhere and import those values into your project. This 
also means that typing d j ango-admi n. py at the command-line works without the — 
setti ngs option. 


For the settings setup that 

we just described, here are the values to use with the — setti ngs com- 

mand line option or the DOANGO_SETTINGS_MODULE environment variahle: 

Environment 

Option To Use With — settings (or 
DJANGO_SETTINGS_MODULE value) 

Your local development 

server 

twoscoops.settings.local 

Your staging server 

twoscoops.settings.staging 

Your test server 

twoscoops.settings.test 

Your production server 

twoscoops.settings.production 

Tahle 5.2: Setting DJANGO_SETTINGS_MODULE per location 
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5.2.1 AD evelopment S ettings Example 

As mentioned earlier, we need settings configured for development, such as selecting the console 
email backend, setting the project to run in DEBUG mode, and setting other configuration options 
that are used solely for development purposes. We place development settings like the foUowing into 
settingsAocal.py. 



Now try it out at the command line with: 



Open http ://127. 0.0. 1:8000 and enjoy your development settings, ready to go into version 
control! You and other developers will he sharing the same development settings files, which for 
shared projects, is awesome. 

Yet theres another advantage: No more‘i f DEBUG’or‘if not DEBUG’logic to copy/paste around 
hetween projects. Settings just got a whole lot simpler! 
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At this point we want to take a moment to note that Django settings files are the single, solitary 
place we advocate using Import *. The reason is thatyor the singular case of Django setting modules 
we want to override ali the namespace. 


5.2.2 Multiple Development Settings 

Sometimes we’re working on a large project where different developers need different settings, and 
sharing the same local.py settings module with teammates won’t do. 

Well, it’s stili better tracking these settings in version control than relying on everyone customizing 
the same local.py module to their own tastes. A nice way to do this is with multiple dev settings files, 
e.g. local_audrey.py and local^ydanny.py. 



Why? It’s not only good to keep all your own settings files in version control, but it’s also good to be 
able to see your teammates’ dev settings files. That way, you can teU if someone’s missing a vital or 
helpful setting in their local development setup, and you can make sure that everyone’s local settings 
files are synchronized. Here is what our projects frequently use for settings layout: 
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5.3 Separate Configuration From Code 

One of the causes of the local_settings anti-pattern is that putting SECRET_KEY, AWS keys, API 

keys, or server-specific values into settings files has problems: 

>• Config varies substantially across deploys, code does not. 

>- Secret keys are configuration values, not code. 

>■ Secrets often should be just that: secret! Keeping them in version control means that everyone 
with repository access has access to them. 

>- Platforms-as-a-service usually doht give you the ability to edit code on individual servers. Even 
if they aUow it, it’s a terribly dangerous practice. 

To resolve this, our answer is to use environment variables in a pattern we like to caU, weU, The 

Environment Variables Pattern. 

Every operating system supported by Django (and Python) provides the easy capability to create 

environment variables. 

Here are the benefits of using environment variables for secret keys: 

>■ Keeping secrets out of settings allows you to store every settings file in version control without 
hesitation. All of your Python code really should be stored in version control, including your 
settings. 

>- Instead of each developer maintaining an easily-outdated, copy-and-pasted version of the 
local_settings.py.example file for their own development purposes, everyone shares the same 
version-controlled settingsAocal.py . 

>■ System administrators can rapidly deploy the project without having to modify files containing 
Python code. 

>- Most platforms-as-a-service recommend the use of environment variables for configuration 
and have built-in features for setting and managing them. 


TIP: 12 Factor App: Store Config in the Environment 


If youVe read the 12 Factor App’s article on configuration youT recognize this pattern. For 
reference, see 12f actor. net/confi g. Some developers even advocate combining the use 
of environment variables with a single settings modules. We cover this practice in Appendix 
E: Settings Alternatives. 
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5.3.1 A Caution Before Using Environment Variables for Secrets 

Before you begin setting environment variables, you should have the foUowing: 

>- A way to manage the secret information you are going to store. 

>- A good understanding of how bash works with environment variables on servers, or a willing- 
ness to have your project hosted by a platform-as-a-service. 

For more information, see en. wi ki pedi a. org/wi ki /Envi ronment_vari able. 


WARNING: Environment Variables Do Not Work With Apache 


If your target production environment uses Apache (outside of Elastic Beanstalk), then you 
wiU discover that setting operating system environment variables as described below doesnt 
Work. Confusing the issue is that Apache has its own environment variable system, which is 
almost but not quite what you’ll need. 

If you are using Apache and want to avoid the local_settings anti-pattern, we recommend 
reading Section 5.4: When You Cant Use Environment Variables later in this chapter. 


5.3.2 How to Set Environment Variables Locally 

On Mac and many Linux distributions that use bash for the sheU, one can add lines like the foUowing 
to the end of a .bashrc, .bash^rofile, or .profile. When dealing with multiple projects using the same 
API but with different keys, you can also place these at the end of your virtualenvs bin/postactivate 
script: 



On Windows Systems, it’s a bit trickier. You can set them one-by-one at the command line (cmd.exe) 
in a persistent way with the setx command, but you’ll have to close and reopen your command 
prompt for them to go into effect. A better way is to place these commands at the end of the vir- 
tualenv’s bin/postactivate.bat script so they are avaUable upon activation: 
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PowerShell is much more powerful than the default Windows shell and comes with Windows Vista 
and above. Setting environment variables while using PowerShell: 

For the current Windows user only: 



Machine-wide: 



For more information on Powershell, see en. wi ki pedi a. org/wi ki / PowerShell 


TIP: virtualenvwrapper Makes This Easier 


Mentioned earlier in this book, virtualenvwrapper, simplifies per-virtualenv environment 
variables. It’s a great tool. Of course, setting it up requires a more-than-basic understanding 
of the shell and Mac OS X, Linux, or Windows. 


5.3.3 How to Unset Environment Variables Locally 

When you set an environment variable via the commands listed above it will remain in existence 
within that terminal shell until it is unset or the shell is ended. This means that even if you deactivate a 
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virtualenv, the environment variable remains. In our experience, this is fine 99% of the time. However, 
there are occasions when we want to tightly control environment variables. To do this, we execute 
the appropriate command for the operating system or sheU variant: 




If you are using virtualenvwrapper and want to unset environment variables whenever a virtualenv is 
deactivated, place these commands in t\\e. postdeactivate script. 


5.3.4 How to Set Environment Variables in Production 

If youre using your own servers, your exact practices will differ depending on the tools youre us¬ 
ing and the complexity of your setup. For the simplest 1-server setup for test projects, you can set 
the environment variables manually. But if youre using Scripts or tools for automated server provi- 
sioning and deployment, your approach may be more complex. Check the documentation for your 
deployment tools for more Information. 

If your Django project is deployed via a platform-as-a-service (PaaS), check the documentation for 
specific instructions. WeVe included instructions here for Elastic Beanstalk and Heroku so that you 
can see that it’s similar for platform-as-a-service options. 



To see how you access environment variables from the Python side, open up a new Python prompt 
and type: 
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To access environment variables from one of your settings files, you can do something like this: 



This snippet simply gets the value of the SOME_SECRET_KEY environment variable from the operat- 
ing System and saves it to a Python variable called SOME_SECRET_KEY. 

Following this pattern means all code can remain in version control, and ali secrets remain safe. 


5.3.5 Handling Missing Secret Key Exceptions 

In the above implementation, if the SECRET_KEY isn’t available, it wiU throw a KeyError , making 
it impossible to start the project. That’s great, but a KeyError doesnt teU you that much about 
whats actuaUy wrong. Without a more helpful error message, this can be hard to debug, especially 
under the pressure of deploying to servers while users are waiting and your ice cream is melting. 

Here’s a useful code snippet that makes it easier to troubleshoot those missing environment variables. 
If youre using our recommended environment variable secrets approach, you’ll want to add this to 
your settings/base.py file: 
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5.J: Separate Configuration From Code 


# into your settings, but ImproperlyConfigured is an exception. 
from django.core.exceptions import ImproperlyConfigured 

def get_env_variable(var_name) : 

"""Get the environment variable or return exception.""" 

try : 

return os . environ[var_name] 
except KeyError: 

error_msg = 'Set the {} environment variable format(var_name) 
raise ImproperlyConfigured(error_msg) 


Then, in any of your settings files, you can load secret keys from environment variables as follows: 



Now, if you don’t have SOME_SECRET_KEY set as an environment variable, you get a traceback 

that ends with a useful error message like this: 



WARNING: Doii’t Import Django Components Into Settings 
Modules 


This can have many unpredictable side effects, so avoid any sort of import of Django com¬ 
ponents into your settings. ImproperlyConfigured is the exception because it’s the official 
Django exception for...weIl...improperly configured projects. And just to be helpful we add 
the name of the problem setting to the error message. 
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PACKAGE TIP: Packages for Settings Management 


A number of third-party packages take the idea of our get_env_variable() function 
and expand on it, including features like defaults and types and supporting .env files. The 
downside is the same you get with any complex packages: sometimes the edge cases cause 
problems. Nevertheless, most of them are quite useful and weVe listed some of our favorites: 
>- github. com/joke2k/dj ango-envi ron (Used in Cookiecutter Django) 

>- github.com/jazzband/django-configurations 


TIP: Using django-admin.py Instead of manage.py 


The oflicial Django documentation says that you should use django-admin.py rather than 

manage.py when working with multiple settings files: 

docs.dj angoproj ect.com/en/1.11/ref/django-admin/ 

That being said, ifyoure struggling with getting django-admin.py to work, it’s perfectly okay 
to develop and launch your site running it with manage.py. 


5.4 When You Can’t Use Environment Variables 

The problem with using environment variables to store secrets is that it doesnt always work. The 
most common scenario for this is when using Apache for serving HTTP, but this also happens even 
in Nginx-based environments where operations wants to do things in a particular way. When this 
occurs, rather than going back to the local_settings anti-pattern, we advocate using non-executable 
files kept out of version control in a method we like to call the secrets file pattern. 

To implement the secrets file pattern, follow these three steps: 

O Create a secrets file using the configuration format of choice, be it JSON, .env, Config, YAML, 
or even XML. 

0 Add a secrets loader (JSON-powered example below) to manage the secrets in a cohesive, 
explicit manner. 

© Add the secrets file name to the project’s .gitignore file. 
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5.4: When You Carit Use Environment Variables 


5.4,1 UsingJSON Files 

Our preference is to use shallowJSON files. The JSON format has the advantage of being the format 
of choice for various Python and non-Python tools. To use the JSON format, first create a secrets.json 
file: 



To use the secrets.json file, add the following code to your base settings module. 
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SECRET_KEY = get_secret( 'SECRET_KEY' ) 


Nowwe are loading secrets from non-executable JSON files instead of from unversioned executable 
code. Hooray! 


5.4,2 Using ,env, Config, YAML, and XML File Formats 

While we prefer the forced simplicity of shallow JSON, others might prefer other file formats. We’ll 
leave it up to the reader to create additional get_secret () alternatives that work with these for¬ 
mats. Just remember to be familiar with things like yaml. saf e_load () and XML bombs. See 
Section 26.10: Defend Against Python Code Injection Attacks. 


5.5 Using Multiple Requirements Files 

Finally, theres one more thing you need to know about multiple settings files setup. It’s good practice 
for each settings file to have its own corresponding requirements file. This means we’re only installing 
what is required on each server. 

To foUow this pattern, recommended to us by Jelf Triplett, first create a requirements/ directory in 
the <repository_root>. Then create ‘.tx/ files that match the contents of your settings directory. The 
results should look something like: 



In the base.txt file, place the dependencies used in aU environments. For example, you might have 
something like the following in there: 
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Example 5.23: requirements/base.txt 


Dj ango==l.11.0 
psycopg2==2.6.2 
dj angorestframework==3.4.0 


Your local.txt file should have dependencies used for local development, such as: 


Example 5.24: requirements/local.txt 


-r base.txt # includes the base.txt requirements file 

coverage==4.2 
dj ango-debug-toolbar==l.5 


The needs of a continuous integration server might prompt the following for a ci.txt file: 


Example 5.25: requirements/ci.txt 


-r base.txt # includes the base.txt requirements file 

coverage==4.2 
dj ango-j enkins==0.19.0 


Production instaUations should be close to what is used in other locations, so production.txt com- 
monlyjust calls base.txt-. 


Example 5.26: requirements/production.txt 


-r base.txt # includes the base.txt requirements file 


5.5.1 Installing From Multiple Requirements Files 


For local development: 
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For production: 



TIP: Pin Requirements Exactly 


All the pip requirements.txt examples in this chapter are explicitly set to a package version. 
This ensures a more stable project. We cover this at length in Section 21.7.2: Step 2: Add 
Package and Version Numher to Your Requirements. 


TIP: Using Multiple Requirements Files With PaaS 


We cover this in Section 30.2.4: Multiple Requirements Files in Multiple Environments 


5.6 Handling F ile Paths in S ettings 

If you switch to the multiple settings setup and get new file path errors to things like templates and 
media, don’t be alarmed. This section will help you resolve these errors. 

We humbly beseech the reader to never hardcode file paths in Django settings files. This is really bad: 


Example 5.29: Never Flardcode File Pythons 

# settings/base.py 

# Configuring MEDIA_R00T 

# D0N’T DO THIS! Hardcoded to just one user's preferences 
MEDIA_R00T = '/Users/pydanny/twoscoops_project/media' 

# Configuring STATIC_R00T 

# D0N’T DO THIS! Hardcoded to just one user's preferences 


58 


















5 . 6: Handling File Paths in Settings 


STATIC_ROOT = '/Users/pydanny/twoscoops_proj ect/collected_static' 

# Configuring STATICFILES_DIRS 

# DON’T DO THIS! Hardcoded to just one user's preferences 
STATICFILES_DIRS = ['/Users/pydanny/twoscoops_project/static'] 

# Configuring TEMPLATES 

# DON’T DO THIS! Hardcoded to just one user's preferences 
TEMPLATES = [ 

{ 

'BACKEND': 'dj ango.template.backends.dj ango.Dj angoTemplates 
DIRS: ['/Users/pydanny/twoscoops_proj ect/templates',] 

}, 

] 


The above code represents a common pitfall called hardcoding. The above code, called a fixed path, 
is bad because as far as you know, pydanny (Daniel Roy Greenfeld) is the only person who has set 
up their computer to match this path structure. Anyone else trying to use this example will see their 
project break, forcing them to either change their directory structure (unlikely) or change the settings 
module to match their preference (causing problems for everyone else including pydanny). 

Dont hardcode your paths! 

To fix the path issue, we dynamically set a project root variable intuitively named BASE_DIR at the 
top of the base settings module. Since BASE_DIR is determined in relation to the location of base.py, 
your project can be run from any location on any development computer or server. 



We find the cleanest way to set a BASE_DIRTike setting is with Pathlib, part of Python since 3.4 
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that does elegant, clean path calculations: 



Ifyou really want to set your BASE_DIR with the Python Standard library’s os. path library, though, 
this is one way to do it in a way that will account for paths: 



60 









5 . 7 ; Summary 



Withyour various path settings dependent on BASE_DIR, your file path settings should work, which 
means your templates and media should be loading without error. 


TIP: How Different Are Your Settings From the Django 
Defaults? 


If you want to know how things in your project differ from Django’s defaults, use the 
di f fsetti ngs management command. 


5.7 Summary 

Remember, everything except for passwords and API keys ought to be tracked in version control. 

Any project thats destined for a real live production server is bound to need multiple settings and 
requirements files. Even beginners to Django need this kind of settings/requirements file setup once 
their projects are ready to leave the original development machine. We provide our solution, as well 
as an Apache-friendly solution since it works well for both beginning and advanced developers. 

Also, if you prefer a different shell than the ones provided, environment variables stiU work. You’ll 
just need to know the syntax for defining them. 

The same thing applies to requirements files. Working with untracked dependency differences in- 
creases risk as much as untracked settings. 


61 











Chapter 5: Settings and Requirements Files 


62 



6 Model Best Practices 


Models are the foundation of most Django projects. Racing to write Django models without thinking 
things through can lead to problems down the road. 


All too frequently we developers rush into adding or modifying models without considering the 
ramifications of what we are doing. The quick fix or sloppy “temporary” design decision that we toss 
into our code hase now can hurt us in the months or years to come, forcing crazy workarounds or 
corrupting existing data. 


So keep this in mind when adding new models in Django or modifying existing ones. Take your time 
to think things through, and design your foundation to he as strong and sound as possible. 


PACKAGE TIP: Our Picks for Working With Models 


Here’s a quick list of the model-related Django packages that we use in practically every 
project. 

>- django-model-utils to handle common patterns like TimeStampedModel. 

>- django-extensions has a powerful management command caUed shell_plus which 
autoloads the model classes for all instaUed apps. The downside of this library is that it 
includes a lot of other functionality which breaks from our preference for smaU, focused 
apps. 
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6.1 Basies 

6.1.1 BreakUp Apps With Too Many Models 

If there are 20+ models in a single app, think about ways to break it down into smaller apps, as it 
probably means your app is doing too much. In practice, we like to lower this number to no more 
than five models per app. 


6.1.2 Be Careful With Model Inheritance 

Model inheritance in Django is a tricky subject. Django provides three ways to do model inheritance: 

abstract base classes, multi-table inheritance, and proxy models. 


WARNING: Django Abstract Base Classes <> Python Abstract Base 
Classes 


Dont confuse Django abstract base classes with the abstract base classes in the Python Stan¬ 
dard librarys abe module, as they have very different purposes and behaviors. 


Here are the pros and cons of the three model inheritance styles. To give a complete comparison, we 
also include the option of using no model inheritance to begin with: 


Model Inheritance Style 

Pros 

Cons 

No model inheritance: if 

Makes it easiest to understand at 

If there are a lot of fields 

models have a common 

a glance how Django models map 

duplicated across models. 

field, give both models that 

to database tables. 

this can be hard to 

field. 


maintain. 

Abstract base classes: tables 

Having the common fields in an 

We cannot use the parent 

are only created for derived 

abstract parent class saves us from 

class in isolation. 

models. 

typing them more than once. 

We don’t get the overhead of extra 
tables and joins that are incurred 
from multi-table inheritance. 
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Multi-table inheritance: 

tables are created for both 
parent and child. An 
implied OneToOneFi eld 
links parent and child. 

Gives each model its own table, 
so that we can query either parent 
or child model. 

Also gives us the ability to get to 
a child object from a parent 
object: parent.child 

Adds substantial overhead 
since each query on a child 
table requires joins with aU 
parent tables. 

We strongly recommend 
against using multi-table 
inheritance. See the 
warning below. 

Proxy models: a table is only 

Allows us to have an alias of a 

We cannot change the 

created for the original 

model with different Python 

modefs fields. 

model. 

behavior. 



Table 6.1: Pros and Cons of the Model Inheritance Styles 


WARNING: Avoid Multi-Table Inheritance 


Multi-table inheritance, sometimes called “concrete inheritance,” is considered by the authors 
and many other developers to be a bad thing. We strongly recommend against using it. We’ll 
go into more detail about this shortly. 


Here are some simple rules of thumb for knowing which type of inheritance to use and when: 

>- If the overlap between models is minimal (e.g. you only have a couple of models that share one 
or two obvious fields), there might not be a need for model inheritance. Just add the fields to 
both models. 

>- If there is enough overlap between models that maintenance of models’ repeated fields causes 
confusion and inadvertent mistakes, then in most cases the code should be refactored so that 
the common fields are in an abstract base model. 

>- Proxy models are an occasionally-useful convenience feature, but they’re very different from 
the other two model inheritance styles. 

>- At all costs, everyone should avoid multi-table inheritance (see warning above) since it adds 
both confusion and substantial overhead. Instead of multi-table inheritance, use explicit One- 
ToOneFi elds and Forei gnKeys between models so you can control whenjoins are traversed. 
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6.1.3 Model Inheritance in Practice: The TimeStampedModel 


It’s very common in Django projects to include a created and modi fied timestamp field on all 
your models. We could manually add those fields to each and every model, but that’s a lot of work 
and adds the riskof human error. Abetter solution is to write a TimeStampedModel to do the work 
for us: 



Take careful note of the very last two lines in the example, which turn our example into an abstract 
base class: 



By defining TimeStampedModel as an abstract base class when we define a new class that inherits 
from it, Django doesnt create a core_timestampedmodel table when migrate is run. 


Let’s put it to the test: 
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This only creates one table: the flavors_flavor database table. That’s exactly the behavior we 
wanted. 

On the other hand, if Ti meStampedModel was not an abstract base class (i.e. a concrete base class 
via multi-table inheritance), it wotdd also create a core_ti mestampedmodel table. Not only that, 
but aUof its subclasses including Flavor would lack the fields and have implicit foreign keys back to 
TlmeStampedModel just to handle created/modl fi ed timestamps. Any reference to Flavor that 
reads or writes to the TlmeStampedHodel would impact two tables. (Thank goodness it’s abstract!) 


Remember, concrete inheritance has the potential to become a nasty performance bottleneck. This 
is even more true when you subclass a concrete model class multiple times. 


Further reading: 


>- docs.djangoproj ect.com/en/1.11/topics/db/models/#model-inherltance 


6.2 Database Migrations 


Django comes with a powerful database change propagation library aptly called “migrations", 
or as we prefer to refer in the book, dj ango. db. mi grati ons. As of Django 1.7 
dj ango. db. mi grati ons replaced the use of the third-party South lihrary, hut as both libraries 
share the same author (Tlndrew Godwin), usage and practices are quite similar. 
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6.2.1 Tips for Creating Migrations 

>- As soon as a new app or model is created, take that extra minute to create the initial 
django.db.migrations for that new model. AU we do is type python manage.py 
makemigrations. 

>■ Examine the generated migration code before you run it, especiaUy when complex changes are 
involved. 7\lso review the SQL that will be used with the sqlmi grate command. 

>- Use the MIGRATION_MODULES setting to manage writing migrations for third-party apps that 
don’t have their own d j ango. db . mi grati ons-style migrations. 

>- Don’t worry about how many migrations are created. If the number of migrations becomes 
unwieldy, use squashmi grati ons to bring them to heel. 

>- Always back up your data before running a migration. 

6.2.2 Adding Python Functions and Custom SQL to Migrations 

d j ango. db. migrati ons can’t anticipate complex changes to your data, or to external components 
that interact with your data. That’s when it’s useful to delve into writing python or custom SQL to 
aid in running migrations. At some point in any project that hits production, you’11 find a reason to 
use either the RunPython or RunSQL classes: 

>- docs.djangoproj ect.com/en/1.11/ref/mi gration-operations/#runpython 
>- docs.dj angoproj ect.com/en/1.11/ref/mi gration-operations/#runsql 

For what it’s worth, our preference is to use RunPython before RunSQL, but we advise sticking to 
where your strengths are. 

6.3 Overcoming Common Obstacles of RunPython 

When we write RunPython-caUed functions, there encounter a few pain points. Most, but not aU of 
these can be resolved. 

6.3.1 Getting Access to a Custom Model Manager’s Methods 

Sometimes you want to be able to filter, exclude, create, or modify records by using custom model 
manager methods. However, by default dj ango. db. mi grati ons excludes these components. For- 
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tunately, we can override this behavior by adding a use_in_migrations = True flag to our custom 

managers. 

See: docs.djangoproj ect.com/en/1.11/topics/migrations/#model-managers 


6.3.2 Getting Access to a Custom Model Method 


Due to how dj ango. db. mi grati ons serializes models, theres no way around this limitation. You 
simply cannot call any custom methods during a migration. See the reference link below: 

docs.dj angoproj ect.com/en/1.11/topics/migrations/#historical-models 


WARNING: Watch Out for Custom Save and Delete Methods 


If you overwrite a models save and delete methods, they won’t be called when caUed by 
RunPython. Consider yourself warned, this can be a devastating gotcha. 


6.3.3 Use RunPython.noop to Do Nothing 


In order for reverse migrations to work, RunPython must be given a reverse_code caUable to undo 

the effects of the code callable. However, some of the code callables that we write are idempotent. 

For example, they combine existing data into a newly added field. Writing a reverse_code callable 

for these functions is either impossible or pointless. When this happens, use RunPython.noop as 
the reverse_code . 

For example, let’s say we create a new model called “Cone”. AU existing scoops need 
their own cone, so we write an add_cones function to add the cones to the database. 
However, when reversing the migration, writing code to remove the cones is pointless; 

migrations.CreateModel.database_backwards will delete the cone.cone table and aU its records 

for us. Therefore, we should use RunPython.noop for the reverse_code : 
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6.3.4 Deployment and Management of Migrations 

>- It goes without saying, but we’U say it anyway: Always back up your data before running a 
migration. 

>• Before deployment, check that you can rollbackmigrations! We eant always have perfect round- 
trips, but not being able to roU back to an earlier state really hurts bug tracking and sometimes 
deployment in larger projects. 

>- If a project has tables with miUions of rows in them, do extensive tests against data of that size 
on staging servers before running a migration on a production server. Migrations on real data 
can take much, much, much more time than anticipated. 

>- If you are using MySQL: 

>- You absolutely positively must back up the database before any schema change. MySQL 
lacks transaction support around schema changes, hence rollbacks are impossible. 

>- If you can, put the project in read-only mode before exeeuting the change. 

>- If not careful, schema changes on heavily populated tables can take a long time. Not 
seconds or minutes, but hours. 



Figure 6.1: Cones migrating south for the winter. Djangos built-in migration system started out as 
an external project called South. 


TIP: Always Put Data Migration Code Into Source Control 


Including migration code in VCS is an absolute necessity. Not including migration code in 
version control is just like not including settings files in VCS: You might be able to develop, 
but should you switch machines or bring someone else into the project, then everything will 
break. 
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6.4 Django Model Design 

One of the most diflicult topics that receives the least amount of attention is how to design good 
Django models. 

How do you design for performance without optimizing prematurely? Let’s explore some strategies 
here. 


6.4,1 Start Normalized 

We suggest that readers of this book need to he familiar with database normalization. If you are 
unfamiliar with datahase normalization, make it your responsihility to gain an understanding, as 
working with models in Django effectively requires a working knowledge of this. Since a detailed 
explanation of the subject is outside the scope of this book, we recommend the following resources: 

>• en.wikipedia.org/wiki /Database_normalization 

>- en.wikibooks.org/wiki/Relational_Database_Design/Normalization 

When youre designingyour Django models, always start off normalized. Take the time to make sure 
that no model should contain data already stored in another model. 

At this stage, use relationship fields liberally. Don’t denormalize prematurely. You want to have a 
good sense of the shape of your data. 


6,4,2 Cache Before Denormalizing 

Often, setting up caching in the right places can save you the trouble of denormalizing your models. 
We’ll cover caching in much more detail in Chapter 24: Finding and Reducing Bottlenecks, so doht 
worry too much about this right now. 


6,4.3 Denormalize Only if AbsolutelyNeeded 

It can be tempting, especially for those new to the concepts of data normalization, to denormalize 
prematurely. Don’t do it! Denormalization may seem like a panacea for what causes problems in a 
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project. However it’s a tricky process that risks adding complexity to your project and dramatically 
raises the risk of losing data. 

Please, please, please explore caching before denormalization. 

When a project has reached the limits of what the techniques descrihed in Chapter 24: Finding and 
Reducing Bottlenecks can address, thats when research into the concepts and patterns of datahase 
denormalization should hegin. 


6.4,4 When to Use Null and Blank 

When defining a model field, you have the ahility to set the null=True and the blank=True op- 
tions. By default, they are False. 

Knowing when to use these options is a common source of confusion for developers. 

WeVe put this guide together to serve as a guide for Standard usage of these model field arguments. 


Field Type Setting nuU=True Setting blank=True 


CharField, TextField, 

SlugField, 

EmaiIField, 

CommaSeparatedlnteger- 

Field, 

UUIDField 

Okay if you also have set hoth 
unique=True and blank=True. 
In this situation, null=True is 
required to avoid unique 
constraint violations when saving 
multiple ohjects with blank 
values. 

Okay if you want the 
corresponding form widget 
to accept empty values. If 
you set this, empty values 
are stored as NULL in the 

datahase if null=True and 

unique=True are also set. 
Otherwise, they get stored 
as empty strings. 

FileField, 

Dont do this. 

Okay. 

ImageField 

Django Stores the path from 

The same pattern for 


MEDIA_ROOT to the file or to the 

CharField applies here. 


image in a CharField, so the same 



pattern applies to FileFields. 
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1 FieldType 

Setting null=True 

Setting blank=True 1 

BooleanField 

Dont do this. Use 

NullBooleanFi eld instead. 

Dont do this. 

IntegerField, 

FloatField, 

DecimalField, 

Durati onFi eld, etc 

Okay if you want to be able to set 
the value to NULL in the database. 

Okay if you want the 
corresponding form widget 
to accept empty values. If 
so, you will also want to set 

null=T rue. 

DateTimeField, 

DateField, TimeField, 

etc. 

Okay if you want to be able to set 
the value to NULL in the database. 

Okay if you want the 
corresponding form widget 
to accept empty values, or if 
you are using auto_now or 
auto_now_add. If it’s the 
former, you will also want 

to set null=True. 

ForeignKey, 

ManyToManyField, 

OneToOneField 

Okay if you want to be able to set 
the value to NULL in the database. 

Okay if you want the 
corresponding form widget 
(e.g. the select box) to 
accept empty values. If so, 
you wiU also want to set 

null=T rue. 

GenericIPAddressField 

Okay if you want to be able to set 
the value to NULL in the database. 

Okay if you want to make 
the corresponding field 
widget accept empty values. 

If so, you will also want to 

set null=True. 

Table 6.2: When to Use Null and Blank by Field 
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Does vanilla 
count as NULL, 
zero,or the 
empty string? 





Figure 6.2: A common source of confusion. 


6.4,5 WhentoUseBinaryField 

This field allows for the storage of raw binary data, or b)^es. We can’t perform filters, excludes, or 
other SQL actions on the field, but there are use cases for it. For example we could store: 

>- MessagePack-formatted content. 

>- Raw sensor data. 

>- Compressed data e.g. the type of data Sentry Stores as a BLOB, but is required to base64- 
encode due to legacy issues. 

The possibilities are endless, but remember that binary data can come in huge chunks, which can slow 
down databases. If this occurs and becomes a bottleneck, the solution might be to save the binary 
data in a file and reference it with a Fi leFi eld. 


WARNING: Don’t Serve Files From BinaryField! 


Storing files in a database field should never happen. If it’s being considered as a solution to 
a problem, find a certified database expert and ask for a second opinion. 

To summarize PostgreSQL expert Frank Wiles on the problems with using a database as a 
file store: 

>- ‘read/write to a DB is always slower than a filesystem’ 

>- ‘your DB backups grow to be huge and more time consuming’ 
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>■ ‘access to the files now requires going through your app (Django) and DB layers’ 

See revsys.com/blog/2012/may/01/three-things-you-should-never-put- 
your-database/ 

When someone thinks there is a good use case for serving files from a database, and quotes a 
success like npmj s . org (stored files in CouchDB), it’s time to do your research. The truth is 
that npmj s .org, migrated its database-as-file-store system to a more traditional file serving 
method years ago. 


6.4,6 Try to Avoid Using Generic Relations 

In generalwe advocate against generic relations and use of models. fi eld . Generi cForei gnKey. 
They are usually more trouble than they are worth. Using them is often a sign that troublesome 
shortcuts are being taken, that the wrong solution is being explored. 

The idea of a generic relations is that we are binding one table to another by way of an unconstrained 
foreign key (Generi cForei gnKey). Using it is akin to using a NoSQL datastore that lacks foreign 
key constraints as the basis for projects that could really use foreign key constraints. This causes the 
following: 

>- Reduction in speed of queries due to lack of indexing between models. 

>- Danger of data corruption as a table can refer to another against a non-existent record. 

The upside of this lack of constraints is that generic relations makes it easier to build apps for things 
that need to interact with numerous model types we might have created. SpecificaUy things like 
favorites, ratings, voting, messages, and tagging apps. Indeed, there are a number of existing apps 
that are built this way. While we hesitate to use them, we are comforted by the fact that the good 
ones are focused on a single task (for example, tagging). 

Over time, weVe found that we can build favorites, ratings, voting, messages, and tagging apps built 
olf ForeignKey and ManyToMany field. For alittle more development work, by avoiding the use of 
Generi cForei gnKey we get the benefit of speed and integrity. 

Where the Generi cForei gnKey becomes really troublesome is when its unconstrained feature be- 
comes the method by which a project’s primary data is defined. For example, if we built an Ice Cream 
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themed project where the relationships between toppings, flavors, containers, orders, and sales were 
ali tracked via Generi c For ei gnKey, we would have the problems described in the bullets above. In 
short: 

>- Try to avoid generic relations and Generi cForei gnKey. 

>- If you think you need generic relations, see if the problem can be solved through better model 
design or the new PostgreSQL fields. 

>• If usage eant be avoided, try to use an existing third-party app. The isolation a third-party app 
provides will help keep data cleaner. 

For another view that shares our opinion, please read lukeplant.me.uk/blog/posts/ 
avoid-dj ango-genericforeignkey 


6.4,7 Make Choices and Sub-Choices Model Constants 

A nice pattern is to add choices as properties to a model. As these are constants tied to your model 
(and the represented data) being able to easily access them everywhere makes development easier. 

This technique is described in https://docs.djangoproject.eom/en/l.ll/ref/models/ 
fi elds/#choi ces. If we translate that to an ice cream-based example, we get: 
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) 

flavor = models.CharField( 
max_length=2, 
choices=FLAVOR_CHOICES 

) 


Using this model, we can do the foUowing: 



This Works in both Python code and templates, and the attribute can be accessed on either the class 
or the instantiated model object. 


6.4,8 Better Model Choice Constants Using Enum 

Our friend Nate Cox recommends using the Enum library to enhance model attributes for choices. 
This Works ifyou are on Python 3.4+ or on Python 2.7 with the pypi . python. org/pypi /enum34 
package installed. 
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@classmethod 

def get_value (cis , member): 

return cis [member].value[0] 

flavor = models.CharField( 
max_length=2, 

choices=[x.value for x in FLAVORS] 


In this situation we’re be able to do: 


Example 6.8: Accessing Enum-Based Choice Model Attributes 


>>> from orders.models import IceCreamOrder 

>>> chocolate = IceCreamOrder.FLAVORS.get_value (' chocolate' ) 

>>> IceCreamOrder.objects.fiIter(flavor=chocolate) 

[<icecreamorder: 35>, <icecreamorder: 42>, <icecreamorder: 49>] 


There are a couple of drawbacks to this, particularly that the filtering itself is a bit more verbose, 
but we gain the benefit of being able to iterate through the possible choices at any given time. Also, 
adding new values is a bit easier. 


6.4,9 PostgreSQL-Specific Fields: When to Use Null and Blank 


FieldType 

Setting null=True 

Setting blank=True 

ArrayField 

Okay. 

Okay. 

HStoreField 

Okay. 

Okay. 

IntegerRangeField, 

BigIntegerRangeField, 
and FloatRangeField 

Okay if you want to be able to set 
the value to NULL in the database. 

Okay if you want the 
corresponding form widget 
to accept empty values. If 
so, you will also want to set 

null=T rue. 
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FieldType Setting null=True Setting blank=True 


Dateti meRangeFi eld if you want to be able to set O^ay if you want the 

and DateRangeFi eld the value to NULL in the database. corresponding form widget 

to accept empty values, or if 
you are using auto_now or 
auto_now_add. If so, you 
will also want to set 
null=T rue. 

ilSONField Okay. Okay. 

Table 6.3: When to Use Null and Blank for Postgres Fields 

6.5 The Model _meta API 

This _meta API is unusual in the following respects: 

>- It is prefixed with yet is a public, documented API. 

>- Unlike other _-prefixed components of Django _meta follows the same deprecation patterns 
as the rest of the framework. 

The reason for this is that before Django 1.8, the model _meta API was unofficial and purposely 
undocumented, as is normal with any API subject to change without notice. The original purpose of 
_meta was simply for Django to store extra info about models for its own use. However, it proved 
so useful that it is now a documented API. 

For most projects you shouldnt need _meta. The main uses for it are when you need to: 

>- Get a list of a models fields. 

>- Get the class of a particular field for a model (or its inheritance chain or other info derived 
from such). 

>- Ensure that how you get this information remains constant across future Django versions. 

Examples of these sorts of situations: 

>- Building a Django model introspection tool. 

>- Building your own custom specialized Django form library. 
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> Creating admin-like tools to edit or interact with Django model data. 

>- Writing visualization or analysis libraries, e.g. analyzing info only about fields that start with 
roo . 

Further reading: 

>- Model _meta docs: docs. dj angoproj ect. com/en/1.11/ ref/models/meta/ 


6.6 Model Managers 

Every time we use the Django ORM to query a model, we are using an interface caUed a model 
manager to interact with the database. Model managers are said to act on the full set of all possible 
instances of this model class (all the data in the table) to restrict the ones you want to work with. 
Django provides a default model manager for each model class, but we can define our own. 

Here’s a simple example of a custom model manager: 



Now, if we first want to display a count of all of the ice cream flavor reviews, and then a count of just 
the published ones, we can do the following: 


81 







Chapter 6: Model Best Practices 



Easy, right? Yet wouldnt it make more sense if you just added a second model manager? That way 
you could have something like: 


Example 6.11: lUusory Benefits of Using Two Model Managers 

>>> from reviews.models import FlavorReview 
>>> FlavorReview.objects.fiIter().count() 

35 

>>> FlavorReview.published.fiIter().count() 

31 


On the surface, replacing the default model manager seems like the obvious thing to do. Unfortu- 
nately, our experiences in real project development makes us very careful when we use this method. 
Why? 

First, when using model inheritance, children of abstract base classes receive their parent’s model 
manager, and children of concrete base classes do not. 

Second, the first manager applied to a model class is the one that Django treats as the default. This 
breaks significantly with the normal Python pattern, causing what can appear to be unpre- 
dictable results from QuerySets. 

With this knowledge in mind, in your model class, obj ects = models. Manager () should be 
defined manually above any custom model manager. 


WARNING: Know the Model Manager Order of Operations 


Always set obj ects = models. Manager () above any custom model manager that has a 
new name. 


Additional reading: docs. djangoproj ect. com/en/1.11/topics/db/managers/ 
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6.7 Understanding Fat Models 

The concept of fat models is that rather than putting data-related code in views and templates, 
instead we encapsulate the logic in model methods, classmethods, properties, even manager methods. 
That way, any view or task can use the same logic. For example, if we have a model that represents 
Ice Cream reviews we might attach to it the following methods: 

>- Review.create_review(cls, user, rating, title, description) A class- 
method for creating reviews. CaUed on the model class itself from HTML and REST views, 
as well as an import tool that accepts spreadsheets. 

>- review.product_average A review instance property that returns the reviewed project’s 
average rating. Used on review detail views so the reader can get a feel for the overaU opinion 
without leaving the page. 

>- review. found_useful(self, user, yes) A method that sets whether or not readers 
found the review useful or not. Used in detail and list views, for both HTML and REST 
implementations. 

As can be inferred from this list, fat models are a great way to improve reuse of code across a project. 
In fact, the practice of moving logic from views and templates to models has been growing across 
projects, frameworks and languages for years. This is a good thing, right? 

Not necessarily. 

The problem with putting all logic into models is it can cause models to explode in size of code, 
becoming what is called a ‘god object’. This anti-pattern results in model classes that are hundreds, 
thousands, even tens of thousands of lines of code. Because of their size and complexity, god objects 
are hard to understand, hence hard to test and maintain. 

When moving logic into models, we try to remember one of the basic ideas of object-oriented pro- 
gramming, that big problems are easier to resolve when broken up into smaller problems. If a model 
starts to become unwieldy in size, we begin isolating code that is prime for reuse across other mod¬ 
els, or whose complexity requires better management. The methods, classmethods, and properties 
are kept, but the logic they contain is moved into Model Behaviors or Stateless Helper Functions. Let’s 
cover both techniques in the following subsections: 
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6.7.1 Model Behaviors a.k.a Mixins 

Model behaviors embrace the idea of compositiori and encapsulation via the use of mixins. Models 
inherit logic from abstract models. For more Information, see the following resources: 

>- blog. kevinastone. com/dj ango-model-behaviors. html Kevin Stones article on us- 
ing composition to reduce replication of code. 

>- medium.com/eshares-blog/superchargi ng-dj ango-producti vity-8dbf9042825e 
Includes a really good section on using DateTimeField for logical deletes. 

>- Section 10.2: Using Mixins With CBVs. 

6.7.2 Stateless Helper Functions 

By moving logic out of models and into utUity functions, it becomes more isolated. This isolation 
makes it easier to write tests for the logic. The downside is that the functions are stateless, hence all 
arguments have to be passed. 

We cover this in Chapter 29: What About Those Random Utilities?. 


6.7.3 Model Behaviors vs Helper Functions 

In our opinion, alone neither of these techniques are perfect. Flowever, when both are used judiciously, 
they can make projects shine. Understanding when to use either isn’t a static Science, it is an evolving 
process. This kind of evolution is tricky, prompting our suggestion to have tests for the components 
of fat models. 


6.8 Summary 

Models are the foundation for most Django projects, so take the time to design them thoughtfully. 

Start normalized, and only denormalize if you ve already explored other options thoroughly. You 
may be able to simplify slow, complex queries by dropping down to raw SQL, or you may be able to 
address your performance issues with caching in the right places. 
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Dont forget to use indexes. Add indexes when you have a better feel for how youre using data 
throughout your project. 

If you decide to use model inheritance, inherit from abstract base classes rather than concrete models. 
You’U save yourself from the confusion of dealing with implicit, unneeded joins. 

Watch out for the “gotchas” when using the null=True and blank=True model field options. Refer 
to our handy table for guidance. 

You may find django-model-utils and django-extensions pretty handy. 

Finally, fat models are a way to encapsulate logic in models, but can aU too readily turn into god 
objects. 

Our next chapter is where we begin talking about queries and the database layer. 
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7 I Queries and the Database Layer 


Most of the queries we write are simple. Djangos Object-Relational Model or ORM provides 
a great productivity shortcut: not only generating decent SQL queries for common use cases, but 
providing model access/update functionality that comes complete with validation and security. It 
allows us to trivially write code that works with different database engines. This feature of ORMs 
powers much of the Django third-party package ecosystem. If you can write your query easily with 
the ORM, then take advantage of it! 

The Django ORM, like any ORM, converts data from different types into objects that we can use 
pretty consistently across supported databases. Then it provides a set of methods for interacting with 
those objects. For the most part, Djangos does a pretty good job at what it’s designed to do. However, 
it does have quirks, and understanding those quirks is part of learning how to use Django. Let’s go 
over some of them, shall we? 

7.1 Use get_object_or_404() for Single Objects 

In views such as detail pages where you want to retrieve a single object and do something with it, use 
get_object_or_404() instead of get(). 


WARNING: get_object_or_404() Is for Views Only 


>- Only use it in views. 

>- Don’t use it in helper functions, forms, model methods or anything that is not a view 
or directly view related. 

Many years ago a certain Python coder who we won’t list by name, but likes to do 
cartwheels, was deploying his first Django project. So entranced was he by Django’s 
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get_obj ect_or_404 () function that he used it everywhere, in views, in models, in forms, 
everywhere. In development this worked great and passed tests. Unfortunately, this uncon- 
strained use meant that when certain records were deleted by the admin staff, the entire site 
broke. 

Keep get_obj ect_or_404 () in your views! 


7.2 Be Careful With Queries That Might Throw 
Exceptions 

When youre getting a single Django model instance with the get_obj ect_or_404 () shortcut, 
you don’t need to wrap it in a try-except block. That’s because get_obj ect_or_404 () already does 
that for you. 

However, in most other situations you need to use a try-except block. Some tips: 


7.2.1 ObjectDoesNotExist vs. DoesNotEjdst 

Obj ectDoesNotExi st can be applied to any model object, whereas DoesNotExi st is for a specific 
model. 
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def list_any_line_item (model, sku): 

try : 

return model.objects.get(sku=sku, quantity_gt=0) 

except ObjectDoesNotExist: 

msg = 'We are out of {0}' .format(sku) 
raise OutOfStock(msg) 


7.2.2 When You Just Want One Object but Get Three Back 


If it’s possible that your query may return more than one object, check for a Multi pleObj ectsRe- 
turned exception. Then in the except clause, you can do whatever makes sense, e.g. raise a special 
exception or log the error. 



7.3 Use Lazy Evaluation to Make Queries Legible 

Djangos ORM is very powerful. And with such power comes the responsibility to make code legible, 
hence maintainable. With complex queries, attempt to avoid chaining too much functionality on a 
small set of lines: 
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Example 7.3: Illegible Queries 

# Don't do this! 

from django-models import Q 

from promos.models import Promo 

def fun_function(name=None): 

"""Find working ice cream promo""" 

# Too much query chaining makes code go off the screen or page. Not good. 
return Promo.objects.acti ve().filter(Q(name_startswith=name)|Q(deseriptior 


This is unpleasant, right? Yet ifwe add in advanced Django ORM tools, then it will go from unpleas- 
ant to as terrible as a sriracha-based ice cream topping. To mitigate this unpleasantness, we can use 
the lazy evaluation feature of Django queries to keep our ORM code clean. 

By lazy evaluation, we mean that the Django ORM doesnt make the SQL calls until the data is 
actually needed. We can chain ORM methods and functions as much as we want, and until we try 
to loop through the resuit, Django doesnt touch the database. Instead ofbeing forced to chain many 
methods and advanced database features on a single line, we can break them up over as many lines 
as needed. This increases readability, which improves the ease of maintenance, which increases time 
for getting ice cream. 

Here we take the code from bad example 7.3 and break it up into more legible code: 
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) 

results = results.exclude(status= 'melted' ) 
results = results.select_related (' flavors' ) 
return results 


As can be seen in the corrected code, we can more easily teli what the end resuit will be. Even better, 
by breaking up the query statement we can comment on specific lines of code. 


7.3.1 Chaining Queries for Legibility 

This technique borrows from the Pandas and JavaScript communities. Instead of using lazy evalua¬ 
tion, it’s possible to chain queries thus: 



The downside to this approach is that debugging isn’t as easy as using the lazy evaluation method of 
writing a query. We simply can’t stick a PDB or IPDB call in the middle of a query defined this way. 
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To get around this, we have to do a bit of commenting out: 



7.4 Lean on Advanced Query Tools 

Djangos ORM is easy to learn, intuitive, and covers many use cases. Yet there are a number of things 
it does not do well. What happens then is after the queryset is returned we begin processing more 
and more data in Python. This is a shame, because every database manages and transforms data faster 
than Python (or Ruby, JavaScript, Go, Java, et al). 

Instead of managing data with Python, we always try to use Djangos advanced query tools to do the 
lifting. In doing so we not only benefit from increased performance, we also enjoy using code that is 
more proven (Django and most databases are constantly tested) than any Python-based workarounds 
we create. 

7.4,1 Query Expressions 

When performing reads on a database, query expressions can be used to create values or computations 
during that read. If that sounds confusing, don’t feel alone, we’re confused too. Since a code example 
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is worth a thousand words, let’s provide an example of how they can benefit us. In our case, we’re 
trying to list all the customers who have on average ordered more than one scoop per visit to an ice 
cream store. 

First, how this might be done, albeit dangerously, without query expressions: 


Example 7.7: No Query Expressions 
# Don't do this! 

from models.customers import Customer 
customers = [] 

for customer in Customer.objects.iterator(): 

if customer.scoops_ordered > customer.store_visits: 
customers.append(customer) 


This example makes us shudder with fear. Why? 

>- It uses Python to loop through all the Customer records in the database, one by one. This is 
slow and memory consuming. 

>- Under any volume of use, it will generate race conditions. This is where while we’re running 
the script, customers are interacting with the data. While probably not an issue in this simple 
‘READ’ example, in real-world code combining that with an ‘UPDATE’ it can lead to loss of 
data. 

Eortunately, through query expressions Django provides a way to make this more efficient and race- 
condition free: 



What this does is use the database itself to perform the comparison. Under the hood, Django is 
running something that probably looks like: 
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Query Expressions should be in your toolkit. They increase the performance and stability of projects. 
>• docs.dj angoproj ect.com/en/1.11/ref/models/expressions/ 


7.4,2 Database Functions 


Since Django 1.8 weve been able to easily use common database functions such as UPPER(), 
LOWERO, COALESCE 0, CONCAT(), LENGTH(), and SUBSTR (). Of all the advanced query 
tools provided by Django, these are our favorites. Why? 

O Very easy to use, either on new projects or existing projects. 

0 Database functions allow us to move some of the logic from Python to the database. This can 
be a performance booster, as processing data in Python is not as fast as processing data in a 
database. 

© Database functions are implemented diiferently per database, but Djangos ORM abstracts 
this away. Code we write using them on PostgreSQL will work on MySQL or SQLiteS. 

O They are also query expressions, which means they foUow a common pattern already established 
by another nice part of the Django ORM. 

Reference: 

>- docs.dj angoproj ect.com/en/1.11/ref/models/database-functions/ 


7.5 Don’t Drop Down to Raw SQL Until It’s Necessary 

Whenever we write raw SQL we lose elements of security and reusability. This does’t just apply to 
internal project code, but also to the rest of the Django world. Specifically, if you ever release one 
of your Django apps as a third-party package, using raw SQL will decrease the portability of the 
work. Also, in the rare event that the data has to be migrated from one database to another, any 
database-specific features that you use in your SQL queries wiU complicate the migration. 
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So when should you actually write raw SQL? If expressing your query as raw SQL would drastically 
simplify your Python code or the SQL generated by the QRM, then go ahead and do it. For example, 
if youre chaining a number of QuerySet operations that each operate on a large data set, there may 
be a more efficient way to write it as raw SQL. 


TIP: Malcolm Tredinnick’s Advice on Writing SQL in Django 


Django core developer Malcolm Tredinnick said (paraphrased): 

“The QRM can do many wonderful things, but sometimes SQL is the right 
answer. The rough policy for the Django QRM is that it’s a storage layer that 
happens to use SQL to implement functionality. If you need to write advanced 
SQL you should write it. I would balance that by cautioning against overuse of 
the raw() andextra() methods.” 


TIP: Jacob Kaplan-Moss’ Advice on Writing SQL in Django 


Django project co-leader Jacob Kaplan-Moss says (paraphrased): 

“If it’s easier to write a query using SQL than Django, then do it. extra () is 
nasty and should be avoided; r aw () is great and should be used where appropri- 
ate. 
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Figure 7.1: This flavor of ice cream contains raw SQL. It’s a bit chewy. 


7.6 Add Indexes as Needed 

While adding db_i ndex=True to any model field is easy, understanding when it should be done 
takes a bit ofjudgment. Our preference is to start without indexes and add them as needed. 

When to considet adding indexes: 

>- The index would be used frequently, as in 10-25% of ali queries. 

>- There is real data, or something that approximates real data, so we can analyze the results of 
indexing. 

>- We can run tests to determine if indexing generates an improvement in results. 

When using PostgreSQL, pg_stat_acti vi ty telis us what indexes are actually being used. 

Once a project goes live, Chapter 24: Finding and Reducing Bottlenecks, has information on index 
analysis. 
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TIP: Class-Based Model Indexes 


Django 1.11 introduces the d j ango. db. models. i ndexes module, the Index 
class, and the Meta, i ndexes option. These make it easy to create all sorts of 
database indexes: just subclass Index, add it to Meta. i ndexes, and youre done! 
dj ango. contri b. postgres. 1 ndexes currently includes Brinlndex and Ginlndex, 
but you can imagine Hashindex, Gi stindex, SpGi stindex, and more. 

>- docs.djangoproject.com/en/1.11/ref/models/indexes/ 

>• docs.djangoproject.com/en/1.11/ref/models/options/#!ndexes 


7.7 Transactions 


The default behavior of the ORM is to autocommit every query when it is caUed. In the case of 
data modification, this means that every time a .create() or .update() is called, it immediately 
modifies data in the SQL database. The advantage of this is that it makes it easier for beginning 
developers to understand the ORM. The disadvantage is that if a view (or some other operation) 
requires two or more database modifications to occur, if one modification succeeds and the other 
fails, the database is at risk of corruption. 


The way to resolve the risk of database corruption is through the use of database transactions. A 
database transaction is where two or more database updates are contained in a single unit ofwork. If 
a single update fails, all the updates in the transaction are rolled back. To make this work, a database 
transaction, by definition, must be atomic, consistent, isolated and durable. Database practitioners 
often refer to these properties of database transactions using the acronym ACID. 


Django has a powerful and relatively easy-to-use transaction mechanism. This makes it much easier to 
lock down database integrity on a project, using decorators and context managers in a rather intuitive 
pattern. 
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7.7.1 Wrapping Each HTTP Request in a Transaction 



Django makes it easy to handle all web requests inside of a transaction with the ATOMIC_REQUESTS 
setting. By setting it to True as shown above, all requests are wrapped in transactions, including 
those that only read data. The advantage of this approach is safety: all database queries in views are 
protected, the disadvantage is performance can suffer. We can’t teli you just how much this will affect 
performance, as it depends on individual database design and how well various database engines 
handle locking. 

WeVe found that this is a great way to ensure at the start that a write-heavy projects database main- 
tains integrity. With lots of traffic, however, weVe had to go back and change things to a more 
focused approach. Depending on the size this can be a small or monumental task. 

Another thing to remember when using ATOMIC_REQUESTS, is that only the database state is rolled 
back on errors. It’s quite embarrassing to send out a confirmation email and then have the transaction 
that wraps a request rolled back. This problem may crop up with any “write” to anything other than 
the database: sending email or SMS, calling a third-party API, writing to the filesystem, etc. There- 
fore, when writing views that create/update/delete records but interact with non-database items, you 
may choose to decorate the view with transacti on. non_atomi c_requests (). 


WARNING: Aymeric Augustin on non_atotnic_ requests () 


Core Django developer and main implementer of the new transaction system, Aymeric Au¬ 
gustin says, “This decorator requires tight coupling between views and models, which wiU 
make a code base harder to maintain. We might have come up with a better design if we 
hadnt had to provide for backwards-compatibility.” 
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Then you can use the more explicit declaration as described below in this super-simple API-style 
fimction-based view: 



Ifyou are using ATOMIC_REQUESTS=True and want to switch to amore focused approachdescribed 
in the following section, we recommend an understanding of Chapter 24: Finding and Reducing 
Bottlenecks, Chapter 22: Testing Stinks and Is a Waste of Money!, and Chapter 32: Continuous 
Integration before you undertake this eifort. 
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TIP: Projects Touching Medical or Financial Data 


For these kinds of projects, engineer systems for eventual consistency rather than for trans- 
actional integrity. In other words, be prepared for transactions to fail and roUbacks to occur. 
Fortunately, because of transactions, even with a roUback, the data wiU remain accurate and 
cie an. 


7.7.2 Explicit Transaction Declaration 

Explicit transaction declaration is one way to increase site performance. In other words, specifying 
which views and business logic are wrapped in transactions and which are not. The downside to this 
approach is that it increases development time. 


TIP: Aymeric Augustin on ATOMIC_REQyESTS vs. Ejcplicit 
Transaction Declaration 


Aymeric Augustin says, ‘Use ATOMIC_REQyESTS as long as the performance overhead 
is bearable. That means “forever” on most sites.’ 


When it comes to transactions, here are some good guidelines to live by: 

>- Database operations that do not modify the database should not be wrapped in transactions. 

>- Database operations that modify the database should be in a transaction. 

>- Special cases including database modifications that require database reads and performance 
considerations can affect the previous two guidelines. 


If that’s not ciear enough, here is a table explaining when different Django ORM calls should be 
wrapped in transactions. 


Purpose 

ORM method 

Generally Use Transactions? 

Create Data 

.createO, .bulk_create(), .get_or_create(). 

/ 

Retrieve Data 

.get(), .filterO, .count(), .iterate(), .exists(), 
.excludeO, .in_bulk, etc. 


Modify Data 

.updateO 

/ 
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Delete Data 

.deleteO 

/ 


Table 7.1: When to Use Transactions 
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Figure 7.2'. Because no one loves ice cream quite like a database. 


We also cover this in Chapter 24: Finding and Reducing Bottlenecks, specifically subsection Sec- 
tion 24.2.4: Switch ATOMIC_REQyESTS to False. 


TIP: Never Wrap Individual ORM Method Calis 


Djangos ORM actually relies on transactions internally to ensure consistency of data. For 
instance, if an update affects multiple tables because of concrete inberitance, Django has that 
wrapped up in transactions. 

Therefore, it is never useful to wrap an individual ORM method [. create (), . update (), 
. delete ()] call in a transaction. Instead, use explicit transactions when you are calling 
several ORM methods in a view, function, or method. 


7.7.3 django.http.StreamingHttpResponse and Transactions 

If a view is returning dj ango. http. Streami ngHttpResponse, it’s impossible to handle trans¬ 
action errors once the response has begun. If your project uses this response method then 
ATOMIC_REQUESTS should do one of the foUowing: 
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O Set ATOMIC_REQUESTS to Django’s default, which is False. Then you can use the techniques 
explored in Section 7.7.2: Explicit Transaction Declaration. Or... 

0 Wrap the view in the django.db. transaction. non_atomic_requests decorator. 

Keep in mind that you can use ATOMIC_REQUESTS with a streaming response, hut the transaction 
will only apply to the view itself. If the generation of the response stream triggers additional SQL 
queries, they will be made in autocommit mode. HopefuUy generating a response doesnt trigger 
database writes... 


7.7.4 Transactions in MySQL 

If the database being used is MySQL, transactions may not be supported depending on your choice 
of table type such as InnoDB or MyISAM. If transactions are not supported, Django wiU always 
function in autocommit mode, regardless of ATOMIC_REQUESTS or code written to support transac¬ 
tions. For more information, we recommend reading the foUowing articles: 

>- docs.djangoproject.com/en/1.11/topics/db/transactions/ 

#transactions-in-mysql 

>- dev.mysql.com/doc/refman/5.0/en/sql-syntax-transactions.html 

7.7.5 Django ORM Transaction Resources 

>- docs. dj angoproj ect. com/en/l. ll/topi cs/db/transactions/ Django’s documen- 
tation on transactions. 

>- Real Python has a great tutorial on the subject of transactions. While written for Django 
1.6, much of the material remains pertinent to this day. realpython.com/blog/python/ 
transaction-management-with-django-1-6 


7.8 Summary 

In this chapter we went over different ways to query a projects persistent data. Now that we know 
how to store data, let’s begin to display it. Starting with the next chapter we’re diving into views! 


102 




8 Function-And Class-Based Views 


Both function-based views (FBVs) and class-based views (CBVs) are in Django 1.11. We recom- 
mend that you understand how to use both types of views. 


TIP: Historical Note 


During the release of Django 1.5, there was a bit of confusion about FBVs due to the wording 
of the release notes and incorrect information on some blog posts. However, no pians ever 
existed for removing function-based views from Django and function-based views remain in 
Django. 


8.1 When to Use FBVs or CBVs 


Whenever you implement a view, think about whether it would make more sense to implement as a 
FBV or as a CBV. Some views are best implemented as CBVs, and others are best implemented as 
FBVs. 


If you arent sure which method to choose, on the next page weVe included a flow chart that might 
be of assistance. 
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Start Here 


Do one of the generic 
class-based views 
closely match what 
you have in mind? 




Can you use a 
class-based view 
just by overriding 
attributes? 




r 


Use a class-based 
view, then! 


What Type of View Shouid You Use? 

A handy flowchart for those moments when you can't decide whether 
to implement a particular view as function-based or class-based. 


Do you need to 
subclass your view to 
create other views? 





Do you have to dive into 
Django s source code 
just to get it working as 
a class-based view? 


Will your view be painfully 
complex to implement 
with a class-based view? 

e.g. Does your view process 
more than one form? 






Use a function-based 
view, then! 


Figure 8.1: Shouid you use a FBV or a CBV? flow chart. 


This flowchart follows our preference for using CBVs over FBVs. We prefer to use CBVs for most 
views, using FBVs to implement only the custom error views or complicated ones that would be a 
pain to implement with CBVs. 


TIP: Alternative Approach - Staying With FBVs 


Some developers prefer to err on the side of using FBVs for most views and CBVs only for 
views that need to be subclassed. That strategy is flne as weU. 
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8.2: Keep View Logic Out ofURLConfs 


8.2 Keep View Logic Out ofURLConfs 

Requests are routed to views via URLConfs, in a module that is normally named urls.py. 
Per Djangos URL design philosophy (docs.djangoproject.eom/en/l.ll/misc/ 
design-philosophies/#url-design), the coupling of views with uris is loose, allows for 
infinite fiexibility, and encourages best practices. 

And yet, this is what Daniel feels like yelling every time he sees complex urls.py files: 

“I didrit write J2EE XML and Eope ZCML configuration files back in the dayjust so you 
darn kids could stick logic into Django uri modules!” 

Remember that Django has a wonderfully simple way of defining URL routes. Like everything else 
we bring up in this book, that simplicity is to be honored and respected. The rules of thumb are 
obvious: 

O The views modules should contain view logic. 

0 The URL modules should contain URL logic. 

Ever see code like this? Perhaps in the official documentation for Class-Based Views? 


Example 8.1: Django CBV-Style URLconf Modules 
# Don't do this! 

from django.conf.uris import uri 

from django.views.generic import DetailView 

from tastings.models import Tasting 

urlpatterns = [ 

url(r'''(?P<pk>\d + )/$' , 

DetaiIView.as_view( 
model=Tasting, 

template_name='tastings/detail.html'), 
name='detai1'), 

url(r'*(?P<pk>\d + )/results/$ ' , 

Detai1View.as_view( 
model=Tasting, 

template_name='tastings/resuits.html'), 
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At a glance this code might seem okay, but we argue that it violates the Django design philosophies: 

>- Loose coupling between views, uris, and models has been replaced with tight coupling, mean- 
ing you can never reuse the view definitions. 

>- Don’t Repeat Yourself is violated by using the same/similar arguments repeatedly between 
CBVs. 

>- Infinite flexibility (for URLs) is destroyed. Class inheritance, the primary advantage of Class 
Based Views, is impossible using this anti-pattern. 

>- Lots of other issues: What happens when you have to add in authentication? And what about 
authorization? Are you going to wrap each URLConf view with two or more decorators? 
Putting your view code into your URLConfs quickly turns your URLConfs into an unmain- 
tainable mess. 

In fact, weve heard from developers that seeing CBVs defined in URLConfs this way was part of 

why they steered ciear of using them. 

Alright, enough griping. We’ll show our preferences in the next section. 

8.3 Stick to Loose Coupling in URLConfs 


Figure 8.2: Loose coupling of chocolate chip cookie dough ice cream. 




Here is how to create URLconfs that avoid the problems we mentioned on the previous page. First, 
we write the views: 
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8.3: Stick to Loose Coupling in URLConfs 



Then we define the uris: 
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regex-r '''(?P<pk>\d + )/ $' , 
view=views.TasteDetailView.as_view(), 
name= 'detai1' 

), 

url( 

regex=r '''(?P<pk>\d + )/results/$' , 
view^views.TasteResultsView.as_view(), 
name= 'results' 

), 

url( 

regex=r '''(?P<pk>\d + )/update/$' , 
view=views.TasteUpdateView.as_view(), 
name= 'update' 

) 

] 


Your first response to our version of this should go something like, “Are you sure this is a good idea? 
You changed things to use two files AND more lines of code! How is this better?" 


Well, this is the way we do it. Here are some of the reasons we find it so useful: 


>- Don’t Repeat Yourself: No argument or attribute is repeated between views. 

>- Loose coupling: WeVe removed the model and template names from the URLConf because 
views should be views and URLConfs should be URLConfs. We should be able to caU our 
views from one or more URLConfs, and our approach lets us do just that. 

>• URLConfs should do one thing and do it weU: Related to our previous bullet, our URLConf 
is now focused primarily on just one thing: URL routing. We arent tracking down view logic 
across both views and URLConfs, we just look in our views. 

>- Our views benefit from being class-based: Our views, by having a formal definition in the 
views module, can inherit from other classes. This means adding authentication, authoriza- 
tion, new content formats, or any other business requirement tossed our way is much easier to 
handle. 

>• Infinite flexibiUty: Our views, by having a formal definition in the views module, can imple- 
ment their own custom logic. 
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8.3.1 What if We Aren’t Using CBVs? 

The same rules apply. 

WeVe encountered debugging nightmares of projects using FBVs with extensive URLConf hack- 

ery, such as elaborate tricks with the_file_attribute of Python modules combined with directory 

walking and regular expressions to automagicaUy create URLConfs. If that sounds painful, it was. 

Keep logic out of URLConfs! 

8.4 Use URL Namespaces 

What URL namespaces do is provide an identifier for app-level and instance level namespaces. URL 
namespaces are one of those things where on the surface they seem like they might not help much, 
but once a developer begins using them they wonder why they didnt use them already We’ll sum up 
using URL namespaces as follows: 

Instead ofwriting URL names like tasti ngs_detai 1 write them like tasti ngs: detai 1 . 

Before we explain why this is so useful, we’ll provide an example of usage based on the app-level 
URLConf code from example 8.2. In the root URLConf we would add: 



To see this in action in a view, let’s take a look at a snippet of code from example 8.1: 
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return reverse( 'tastings:detai1' , 
kwargs={ 'pk' : self .object.pk}) 


See this in action in an HTML template: 



Now that we understand how to implement URL namespaces, let’s cover why they are useful. 


8.4,1 Makes for Shorter, More Obvious and Don’t Repeat Yourself URL 
Names 

In example 8.2 what we don’t see are URL names like “tasti ngs_detai 1” and 
“tastings_results” that copy the model or app name. Instead there are simple, obvious 
names like “detair and ‘‘‘'resulti'. This greatly increases the legibility of apps, especially to newer 
Django developers. 

Also, who wants to type “tastings” or whatever an app is called so many extra times? 
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8.4,2 Increases Interoperability With Third-Party Libraries 

One of the problems ofwriting URL names things like <myapp>_detai 1 is when app names col¬ 
lide. While this might not be a problem with things like our tastings app, it’s certainly happened to 
the authors with blog and contact applications. Fortunately, URL namespaces makes this easy to 
resolve. Assuming that we have an existing contact app, but needed to add a second one, using URL 
namespaces we could integrate them to our root URLConf like so: 



Then work them into our templates doing the following: 
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8.4,3 Easier Searches, Upgrades, and Refactors 

Considering the prevalence of underscores in names for PEP 8-friendly frameworlcs like Django, 
searching code or names like “tasti ngs_detai 1” can be challenging. When a resuit comes up, is 
that for a view name, a URL name, or something else? 

On the other hand, searching for “tasti ngs: detai 1” makes for obvious search resuit responses. 
This can and has made upgrades and refactoring of apps and projects easier, including when interact- 
ing with new third-party libraries. 


8,4,4 Allows for More App and Template Reverse Tricks 

We’re not going to cover any tricks here, because we feel such things are almost never justified. In fact, 
they usuaUy just add to the complexity of a project without adding any tangible benefit. However, 
there are a couple use cases worth mentioning: 

>- Development tools like django-debug-toolbar that perform debug-level introspection. 

>- Projects that aUow end-users to add “modules” to change or alter the behavior of their account. 

While developers can use either of these to justify the use of Creative URL namespaces tricks, as 
always, we recommend trying the simplest approach first. 

8.5 Try to Keep Business Logic Out of Views 

In the past, we ve placed an amazing amount of sophisticated business logic into our views. Unfortu- 
nately, when it became time to generate PDFs, add a REST API, or serve out other formats, placing 
so much logic in our views made it much harder to deliver new formats. 

This is where our preferred approach of model methods, manager methods, or general utility helper 
fimctions come into play. When business logic is placed into easily reusable components, and called 
from within views, it makes extending components of the project to do more things much easier. 

Since it’s not always possible to do this at the beginning of a project, our rule of thumb has become 
whenever we find ourselves duplicating business logic instead of Django boilerplate between views, 
it’s time to move code out of the view. 
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8.6 Django Views Are Functions 

When it comes down to it, every Django view is a function. This flinction takes an HTTP request 
object and turns it into a HTTP response object. If you know anything about basic mathematical 
functions, this process of change should look very familiar. 



This concept of change serves as a Foundation for all sorts of things you can do with Django views, 
be they function- or class-based. 


TIP: Class-Based Views Are Actually Called as Functions 


Djangos CBVs appear to be very different than FBVs. However, the View.as_view() 
classmethod called in URLConfs is actually returning a callable instance of the view. In 
other words, a callback function that handles the request/response cycle in exactly the same 
manner as a function-based view! 


8.6.1 The Simplest Views 

With this in mind, it’s good to remember the simplest possible views that can be created with Django: 
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# The simplest FBV 

def simplest_view(request) : 

# Business logic goes here 
return HttpResponse( 'FBV' ) 

# The simplest CBV 
class SimplestView(View) : 

def get(self, request, *args, **kwargs): 
# Business logic goes here 
return HttpResponse (' CBV' ) 


Why is this useflil to know? 

>- Sometimes we need one-off views that do tiny things. 

>■ Understanding the simplest Django views means we better understand what they are really 
doing. 

>- Illustrates how Django FBVs are HTTP method neutral, but Django CBVs require specific 
HTTP method declaration. 


8.7 Don’t Use locals () as Views Context 

Returning locals () from any callable is an anti-pattern. While it seems like a handy shortcut, in 
fact it is a time consuming nightmare. Let’s use an example to explore why. Here is a view following 
this anti-pattern: 


Example 8.11: Innappropriate Use of localsQ 
# Don't do this! 

def ice_cream_store_display(request, store_id): 
store = get_object_or_404(Store, id=store_id) 
date = timezone.now() 

return render(request, 'melted_ice_cream_report.html', localsO) 


On the surface everything seems fine. 

However, as weVe gone from an explicit design to implicit anti-pattern has made this simple view 
annoying to maintain. Specifically that we don’t know what the view is supposed to return. This 
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becomes an issue because changing any of the variables returned by the view is not immediately 
apparent: 


Example 8.12: Altered Version of Previous Code Example 
# Don't do this! 

def ice_cream_store_disp'Lay(request, store_id): 
store = get_object_or_404(Store, id=store_id) 
now = timezone.now() 

return render(request, 'melted_ice_cream_report.html', tocalsO) 


EIow long did it take you to spot the difference between Bad Example 8.11 and Bad Example 8.12? 
This was a simple example, imagine a more complicated one with a large template. This is why we 
strongly advocate use of explicit context in views: 



>- 7\lex Martellis Reasoning: stackoverflow. com/a/1901720 


8.8 Summary 

This chapter started with discussing when to use either EBVs or CBVs, and matched our own pref- 
erence for the latter. In fact, in the next chapter we’Il start to dig deep into the functionality that can 
be exploited when using EBVs, followed up by a chapter on CBVs. 

We also discussed keeping view logic out of the URLConfs. We feel view code belongs in the apps’ 
views.py modules, and URLConf code belongs in the apps’ urls.py modules. Adhering to this prac- 
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tice allows for object inheritance when used with class-based views, easier code reuse, and greater 
flexibility of design. 
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Best Practices for Function-Based Views 


Since the beginning of the Django project, fiinction-based views have been in frequent use by de- 
velopers around the world. While class-based views have risen in usage, the simplicity of using a 
function is appealing to both new and experienced developers alike. While the authors are in the 
camp of preferring CBVs, we work on projects that use FBVs and here are some patterns weVe 
grown to enjoy. 

9.1 Advantages of FBVs 

The simplicity of FBVs comes at the expense of code reuse: FBVs dont have the same ability to 
inherit from superclasses the way that CBVs do. They do have the advantage of being more obviously 
functional in nature, which lends itself to a number of interesting strategies. 

We follow these guidelines when writing FBVs: 

>- Less view code is better. 

>- Never repeat code in views. 

>- Views should handle presentation logic. Try to keep business logic in models when possible, 
or in forms if you must. 

>- Keep your views simple. 

>- Use them to write custom 403, 404, and 500 error handlers. 

>- Complex nested-if blocks are to be avoided. 

9.2 Passing the HttpRequest Object 

There are times where we want to reuse code in views, but not tie it into global actions such as 
middleware or context processors. Starting in the introduction of this book, we advised creating 
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utility functions that can be used across the project. 

For many utility functions, we are taking an attribute or attributes from the 
dj ango. http. HttpRequest (or HttpRequest for short) object and gathering data or per- 
forming operations. What we ve found is by having the request object itself as a primary argument, 
we have simpler arguments on more methods. This means less cognitive overload of managing 
function/method arguments: just pass in the HttpRequest object! 



The check_spri nkle_ri ghts () function does a quick check against the rights of the user, raising 
a dj ango. core. excepti ons. Permi SS 1 onDeni ed exception, which triggers a custom HTTP 
403 view as we describe in Section 29.4.3: django.core.exceptions.PermissionDenied. 

YouT note that we return back a HttpRequest object rather than an arbitrary value or even a None 
object. We do this because as Python is a dynamically typed language, we can attach additional 
attributes to the HttpRequest. For example: 
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request.can_sprinkle = True 
return request 

# Return a HTTP 403 back to the user 
raise PermissionDenied 


There’s another reason, which we’ll cover shortly. In the meantime, let’s demonstrate this code in 
actio n: 


Example 9.3: Passing the Request Object in FBVs 


# sprinkles/views.py 

from d jango.shortcuts "import get_object_or_404 
from d jango.shortcuts import render 

from .models import Sprinkle 
from .utils import check_sprinkles 

def s prinkTe_list (request): 

"""Standard list view""" 

request = check_sprinkles(request) 

return render(request, 

"sprinkles/sprinkle_list.html" , 

{"sprinkles" : Sprinkle.objects.all()}) 

def s prinkle_detail (request, pk): 

"""Standard detail view""" 

request = check_sprinkles(request) 

sprinkle = get_object_or_404(Sprinkle, pk=pk) 

return render(request, "sprinkles/sprinkle_detail.html" , 
{"sprinkle": sprinkle}) 

def s prinkle_preview( request): 
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"""Preview of new sprinkle, but without the 
check_sprinkles function being used. 

II II II 

sprinkle = Sprinkle.objects.ali() 
return render(request, 

"sprinkles/sprinkle_preview.html" , 
{"sprinkle": sprinkle}) 


Another good feature about this approach is that it’s trivial to integrate into class-based views: 



TIP: Specific Function Arguments Have Their Place 


The downside to single argument functions is that specific function arguments like ‘pk’, ‘fiavor’ 
or ‘text’ make it easier to understand the purpose of a function at a glance. In other words, 
try to use this technique for actions that are as generic as possible. 


Since we’re repeatedly reusing functions inside functions, wouldnt it be nice to easily recognize when 
this is being done? This is when we bring decorators into play. 
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9.3 Decorators Are Sweet 

For once, this isn’t about ice cream, it’s about code! In computer Science parlance, syntactic sugar is 
a syntax added to a programming language in order to make things easier to read or to express. In 
Python, decorators are a feature added not out of necessity, but in order to make code cleaner and 
sweeter for humans to read. So yes, Decorators Are Sweet. 

When we combine the power of simple functions with the syntactic sugar of decorators, we get handy, 
reusable tools like the extremely useful to the point of being ubiquitous 
dj ango. contri b. auth. decorators. logi n_requi red decorator. 

Here’s a sample decorator template for use in function-based views: 



That might not make too much sense, so we’ll go through it step-by-step, using in-line code com- 
ments to clarify what we are doing. First, let’s modify the decorator template from the previous 
example to match our needs: 
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def check_sprinkles (view_func) : 

"""Check if a user can add sprinkles""" 

@wraps (view_func) 

def new_view_func (request, *args, **kwargs): 

# Act on the request object with utiIs.can_sprinkle() 
request = utiIs.can_sprinkle(request) 

# Call the view function 

response = view_func(request, *args, **kwargs) 

# Return the HttpResponse object 
return response 

return new_view_func 


Then we attach it to the function thus: 
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Figure 9.1: If you look at sprinkles closely, you’ll see that theyre Python decorators. 


TIP: What About functools.wraps()? 


Astute readers may have noticed that our decorator examples used the functools. wraps () 
decorator function from the Python Standard lihrary. This is a convenience tool that copies 
over metadata including critical data like docstrings to the newly decorated function. It’s not 
necessary, but it makes project maintenance much easier. 


9.3.1 Be Conservative With Decorators 

As with any powerful tool, decorators can be used the wrong way. Too many decorators can 
create their own form of obfuscation, making even complex class-based view hierarchies seem 
simple in comparison. When using decorators, establish a limit of how many decorators can 
be set on a view and stick with it. Video on the subject: pyvideo.org/pycon-us-2011/ 
pycon-2011--how-to-write-obfuscated-python.html 


9.3.2 Additional Resources on Decorators 

>- Decorators Explained: 

j effknupp.com/blog/2013/11/29/ 1 mprove-your-python-decorators-explained/ 
>- Decorators and Functional Python: 

brianholdefehr.com/decorators-and-functional-python 
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>■ Decorator Cheat Sheet by author Daniel Roy Greenfeld 
pydanny.com/python-decorator-cheatsheet.html 


9.4 Passing the HttpResponse Object 

Just as with the HttpRequest object, we can also pass around the HttpResponse object from 
function to function. Think of this as a selective Middleware. process_template_response() 
method. See docs.dj angoproj ect.com/en/1.11/topics/http/middleware/ 

#process-template-response. 

Yes, this technique can be leveraged with decorators. See Example 8.5 which gives a hint as to how 
this can be accomplished. 

9.5 Summary 

Function-based views are stili alive and well in the Django world. If we remember that every func¬ 
tion accepts a HttpRequest object and returns an HttpResponse object, we can use that to our 
advantage. We can leverage in generic HttpRequest and HttpResponse altering functions, which 
can also be used to construet decorator functions. 

We’ll close this chapter by acknowledging that every lesson weve learned about function-based views 
can be applied to what we begin to discuss next chapter, class-based views. 
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Django provides a Standard way to write class-based views (CBVs). In fact, as we mentioned in 
previous chapters, a Django view is just a callable that accepts a request object and returns a re¬ 
sponse. For function-based views (FBVs), the view function is that callable. For CBVs, the view class 
provides an as_view() class method that returns the callable. This mechanism is implemented in 
django. VIews. generic . View. All CBVs should inherit from that class, directly or indirectly. 

Django also provides a series of generic class-based views (GCBVs) that implement common patterns 
found in most web projects and illustrate the power of CBVs. 


PACKAGE TIP: Filling the Missing Parts of Django GCBVs 


Out of the box, Django does not provide some very useful mixins for GCBVs. The django- 
braces library addresses most of these issues. It provides a set of clearly coded mixins that 
make Django GCBVs much easier and faster to implement. The library is so useful that many 
of its mixins have been copied into core Django. 


10.1 Guidelines When Working With CBVs 

>- Less view code is better. 

>• Never repeat code in views. 

>- Views should handle presentation logic. Try to keep business logic in models when possible, 
or in forms if you must. 

>- Keep your views simple. 

>■ Keep your mixins simpler. 
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TIP: Familarize YourselfWith ccbv.co.uk 


Arguably this should be placed as the sixth guideline, ccbv. co. uk is so useful that we felt 
it deserved its own tipbox. This site takes all the attributes and methods that every CBV 
defines or inherits and flattens it into one comprehensive page per view. Most Django devel- 
opers, once they get past the tutorials on CBVs, rely on ccbv .co.uk more than the oflicial 
documentation. 


10.2 Using Mixins With CBVs 


Think of mixins in programming along the lines of mixins in ice cream: you can enhance any ice 
cream flavor by mixing in crunchy candy bits, sliced fruit, or even bacon. 
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Figure 10.1: Popular and unpopular mixins used in ice cream. 


Soft serve ice cream greatly benefits from mixins: ordinary vanilla soft serve turns into birthday cake 
ice cream when sprinkles, blue buttercream icing, and chunks of yellow cake are mixed in. 

In programming, a mixin is a class that provides functionality to be inherited, but isn’t meant for 
instantiation on its own. In programming languages with multiple inheritance, mixins can be used 
to add enhanced functionality and behavior to classes. 
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We can use the power of mixins to compose our own view classes for our Django apps. 


When using mixins to compose our own view classes, we recommend these rules of inheritance 
provided by Kenneth Love. The rules follow Pythons method resolution order, which in the most 
simplistic definition possible, proceeds from left to right: 

O The base view classes provided by Django always go to the right. 

0 Mixins go to the left of the base view. 

© Mixins should inberit from Pythons built-in object type. Keep your inheritance chain simple! 
Example of the rules in action: 



In our rather siUy example, the FruityFlavorView class inherits from both FreshFruitMixin 
and TemplateVi ew. 

Since TemplateVi ew is the base view class provided by Django, it goes on the far right (rule 1), 
and to its left we place the FreshFruitMixin (rule 2). This way we know that our methods and 
properties will execute correctly. 

Finally, FreshFruitMixin inherits from object (rule 3). 
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10.3 Which Django GCBV Should Be Used for What Task? 

The power of generic class-based views comes at the expense of simplicity: GCBVs come with a 
complex inheritance chain that can have up to eight superclasses on import. Trying to work out 
exactly which view to use or which method to customize can be very challenging at times. 


To mitigate this challenge, heres a handy chart listing the name and purpose of each Django CBV. 
All views listed here are assumed to be prefixed with django . vi ews . generic. 


Name 

Purpose 

Two Scoops Example 




Vi ew 

Base view or handy view that can be 
used for anything. 

See Section 10.6: Using Just 
dj ango.views .generic. View. 

RedirectView 

Redirect user to another URL 

Send users who visit Vlog-in/’ to 
Vlogin/’. 

TemplateView 

Display a Django HTML template. 

The Vabout/’ page of our site. 

ListView 

List objects 

List of ice cream flavors. 

DetaiIView 

Display an object 

Details on an ice cream flavor. 

FormView 

Submit a form 

The site’s contact or email form. 

CreateView 

Create an object 

Create a new ice cream flavor. 

UpdateView 

Update an object 

Update an existing ice cream 
flavor. 

DeleteView 

Delete an object 

Delete an unpleasant ice cream 
flavor like Vanilla Steak. 

Generic date views 

For display of objects that occur over a 
range of time. 

Blogs are a common reason to 
use them. For Two Scoops, we 
could create a public history of 
when flavors have been added to 

the database. 


Table 10.1: Django CBVUsage 

Table 
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TIP: The Three Schools of Django CBV/GCBVUsage 


WeVe found that there are three major schools of thought around CBV and GCBV usage. 
They are: 

The School of “Use ali the generic views!” 

This school of thought is hased on the idea that since Django provides functionality to reduce 
your workload, why not use that functionality? We tend to helong to this school of thought, 
and have used it to great success, rapidly huilding and then maintaining a numher of projects. 

The School of “Just use django.views.generic.View” 

This school of thought is hased on the idea that the hase Django CBV does just enough and 
is ‘the True CBV, everything else is a Ceneric CBV’. In the past year, we’ve found this can 
be a really useful approach for tricky tasks for which the resource-based approach of “Use aU 
the views” breaks down. We’ll cover some use cases for it in this chapter. 

The School of “Avoid them unless you’re actually suhclassing views” 

Jacob Kaplan-Moss says, “My general advice is to start with function views since they’re 
easier to read and understand, and only use CBVs where you need them. Where do you 
need them? Any place where you need a fair chunk of code to be reused among multiple 
views.” 

We generally helong to the first school, but it’s good for you to know that there’s no real 
consensus on best practices here. 


10.4 General Tips for Django CBVs 


This section covers useful tips for all or many Django CBV and CCBV implementations. We’ve 
found they expedite writing of views, templates, and their tests. These techniques wiU work with 
Class-Based Views or Generic Class-Based Views. As always for CBVs in Django, they rely on 
object oriented programming techniques. 
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10.4,1 Constraining Django CBV/GCBV Access to Authenticated Users 

While the Django CBV documentation gives a helpful working example of using the 
dj ango. contri b. auth. decorators. logi n_requi red decorator with a CBV, but this ex¬ 
ample violates the rule of keeping logic out of urls.py. docs.djangoproject.com/en/1.11/ 
topics/class-based-views/intro/#decorating-class-based-views. 

Fortunately, Django provides a ready implementation of a Logi nRequi redMi xi n object that you 
can attach in moments. For example, we could do the following in all of the Django GCBVs that 
weVe written so far: 



TIP: Don’t Forget the GCBVMixin Order! 


Remember that: 

> Logi nRequi redMi xi n must always go on the far left side. 

>■ The base view class must always go on the far right side. 

If you forget and switch the order, you will get broken or unpredictable results. 


WARNING: Overriding dispatch() When Using LoginRequiredMixin 


Ifyou use Logi nRequi redMi xi n and override the dispatch method, make sure that the first 
thingyou do is call super (FlavorDetaiIvi ew, self). di spatch(request, *args, 
**kwargs) . Any code before the super () call is executed even if the user is not authenti¬ 
cated. 
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10.4,2 Performing Custom Actions on Views With Valid Forms 

When you need to perform a custom action on a viewwith a valid form, the form_vali d () method 
is where the GCBV workflow sends the request. 



To perform custom logic on form data that has already been validated, simply 
add the logic to form_valid (). The return value of form_valid() should be a 
django.http.HttpResponseRedirect. 

10,4,3 Performing Custom Actions on Views With Invalid Forms 

When you need to perform a custom action on a view with an invalid form, the f orm_i nvali d () 
method is where the Django GCBV workflow sends the request. This method should return a 
django. http.HttpResponse. 
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class FlavorCreateView( LoginRequiredMixin, CreateView): 
model = Flavor 

def form_invalid (self , form): 

# Do custom logic here 

return super (FlavorCreateView, self ).form_invalid(form) 


Just as you can add logic to form_valid (), you can also add logic to form_invalid (). 

You’ll see an example of overriding both of these methods in Section 12.5.1: ModelForm Data Is 
Saved to the Form, Then the Model Instance. 




iwHeRiTAis;ct 




Figure 10.2: The other CBV: class-based vanilla ice cream. 


10.4,4 Using the View Object 


If you are using class-based views for rendering content, consider using the view object itself to 
provide access to properties and methods that can be called by other method and properties. They 
can also be called from templates. For example: 
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The nice thing about this is the vs.nousfavors/ app templates can now access this property: 
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10.5 How GCBVs and Forms Fit Together 

A common source of confusion with GCBVs is their usage with Django forms. 

Using our favorite example of the ice cream flavor tracking app, let’s chart out a couple of examples 
of how form-related views might fit together. 

First, let’s define a flavor model to use in this sections view examples: 
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scoops_remaining = models.IntegerField(choices=STATUS_CHOICES, 
default=STATUS_0) 

def get_absolute_url(self) : 

return reverse ("flavors:detai1" , kwargs={"slug" : self.slug}) 


Now, let’s explore some common Django form scenarios that most Django users run into at one 
point or another. 


10.5.1 Views + ModelForm Example 

This is the simplest and most common Django form scenario. Typically when you create a model, 
you want to be able to add new records and update existing records that correspond to the model. 

In this example, we’ll show you how to construet a set of views that will create, update and display 
Flavor records. We’ll also demonstrate how to provide confirmation of changes. 

Here we have the foUowing views: 

O FlavorCreateView corresponds to a form for adding new flavors. 

0 FlavorUpdateView corresponds to a form for editing existing flavors. 

© FlavorDetailView corresponds to the confirmation page for both flavor creation and flavor 
updates. 

To visualize our views: 

FlavorCreateView ► 

(CreateView) 

FlavorUpdateView ► 

(UpdateView) 

Figure 10.3: Views + ModelForm Flow 


FlavorDetailView 

(DetailView) 


FlavorDetailView 

(DetailView) 
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Note that we stick as closely as possible to Django naming conventions. FlavorCreateVi ew sub¬ 
classes Django’s CreateView, FlavorUpdateView subclasses Django’s UpdateView, and Fla- 
vorDetai IVi ew subclasses Django’s Detai IVi ew. 

Writing these views is easy, since it’s mostly a matter of using what Django gives us: 



Simple at first glance, right? We accomplish so much with just a little bit of code! 

But wait, there’s a catch. If we wire these views into a urls.py module and create the necessary tem- 
plates, we’ll uncover a problem: 

The FlavorDetailViewis not a confirmation page. 

For now, that statement is correct. Fortunately, we can fix it quickly with a few modifications to 
existing views and templates. 

The first step in the fix is to use dj ango. contri b . messages to inform the user visiting the Fla- 
vorDetai IVi ew that they just added or updated the fiavor. 
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We’ll need to override the FlavorCreateVi ew. form_valid () and FlavorUpdate- 
View. form_valid 0 methods. We can do this conveniently for both views with a Flavo- 
rActionMixin. 

For the confirmation page fix, we c)\&nge. flavors/views.py to contain the following: 
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Earlier in this chapter, we covered a simpler example of how to override form_valid() within a 
GCBV. Here, we reuse a similar form_vali d () override method by creating a mixin to inherit from 
in multiple views. 

Now we’re using Django’s messages framework to display confirmation messages to the user upon 
every successful add or edit. We define a FlavorActi onMixin whose job is to queue up a confir¬ 
mation message corresponding to the action performed in a view. 


TIP: Mixins Should Inherit From Object 


Please take notice that the FlavorActi onMixin inherits from Pythons object type rather 
than a pre-existing mixin or view. It’s important that mixins have as shallow inheritance chain 
as possible. Simplicity is a virtue! 


After a flavor is created or updated, a list of messages is passed to the context of the FlavorDetai 1- 
Vi ew. We can see these messages ifwe add the following code to the views’ template and then create 
or update a flavor: 



TIP: Reuse the Messages Template Code! 


It is common practice to put the above code into your project’s base HTML template. Doing 
this allows message support for templates in your project. 
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To recap, this example demonstratedyet again howto override the form_valid () method, incorpo¬ 
rate this into a mixin, how to incorporate multiple mixins into a view, and gave a quick introduction 
to the very useful dj ango. contri b. messages framework. 

10.5.2 Views + Form Example 

Sometimes you want to use a Django Form rather than a ModelForm. Search forms are a particularly 
good use case for this, but you’ll run into other scenarios where this is true as well. 

In this example, we’ll create a simple flavor search form. This involves creating a HTML form that 
doesht modify any flavor data. The forms action will query the ORM, and the records found will be 
listed on a search results page. 

Our intention is that when using our flavor search page, if users do a flavor search for “Dough”, 
they should be sent to a page listing ice cream flavors like “Chocolate Chip Cookie Dough,” “Fudge 
Brownie Dough,” “Peanut Butter Cookie Dough,” and other flavors containing the string “Dough” 
in their title. Mmm, we deflnitely want this feature in our web application. 

There are more complex ways to implement this, but for our simple use case, all we need is a single 
view. WeTl use a Flavor Li stVi ew for both the search page and the search results page. 

Here’s an overview of our implementation: 

FlavorListView ► 

(ListView) 

Figure 10.4: Views + Form Flow 


FlavorListView 

(ListView) 


In this scenario, we want to foUow the Standard internet convention for search pages, where ‘q’ is 
used for the search query parameter. We also want to accept a GET request rather than a POST 
request, which is unusual for forms but perfectly flne for this use case. Remember, this form doesht 
add, edit, or delete objects, so we doht need a POST request here. 

To return matching search results based on the search query, we need to modify the Standard queryset 
supplied by the ListView. To do this, we override the ListView' s get_queryset () method. 
We add the following code to flavors/views.py. 
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Now, instead of listing all of the flavors, we list only the flavors whose tities contain the search string. 


As we mentioned, search forms are unusual in that unlike nearly every other HTML form they 
specify a GET request in the HTML form. This is because search forms are not changing data, but 
simply retrieving information from the server. The search form should look something like this: 
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TIP: Specify the Form Target in Search Forms 


We also take care to specify the URL in the form action, because weVe found that search 
forms are often included in several pages. This is why we prefix them with and create 
them in such a way as to be included in other templates. 


Once we get past overriding the Li stVi ew' s get_queryset () method, the rest of this example 
is just a simple HTML form. We like this kind of simplicity. 

10.6 Using Just django.views.generic.View 

It’s entirely possible to build a project just using django. vi ews. generi c. Vi ew for ali the views. 
It’s not as extreme as one might think. For example, if we look at the oflicial Django docu- 
mentatiohs introduction to class-based views (docs.djangoproject.com/en/1.11/topics/ 
class-based-views/intro/#using-class-based-views), we can see the approach is very 
close to how function-based views are written. In fact, we highlighted this two chapters ago in Sec- 
tion 8.6.1: The Simplest Views because it’s important. 

Imagine instead ofwriting function-based views with nested-ifs representing different HTTP meth- 
ods or class-based views where the HTTP methods are hidden behind get_context_data () and 
form_valid () methods, they are readily accessible to developers. Imagine something like: 
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return render(request, 

" flavo rs/flavo r_deta"i l. html" , 

{"flavor": flavor} 

) 

def post(self, request, *args, **kwargs): 

# Handles updates of the Flavor object 
flavor = get_object_or_404(Flavor, slug=kwargs[ 'slug' ]) 
form = FlavorForm(request.POST) 
if form.is_valid() : 
form.save() 

return redirect ("flavors:detai1" , flavor.slug) 


While we can do this in a function-based view, it can be argued that the GET/POST method dec- 
larations within the FlavorVi ew are easier to read than the traditional “i f request. method == 
...” conditions. In addition, since the inheritance chain is so shallow, it means using mixins doesnt 
threaten us with cognitive overload. 

What we find really useful, even on projects which use a lot of generic class-based views, is using 
the d j ango . vi ews. generi c. Vi ew class with a GET method for displaying JSON, PDF or other 
non-HTML content. All the tricks that weVe used for rendering CSV, Excel, and PDF files in 
fimction-based views apply when using the GET method. For example: 
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flavor = get_object_or_404(Flavor, slug=kwargs[ 'slug' ]) 

# create the response 

response = HttpResponse(content_type= 'application/pdf' ) 

# generate the PDF stream and attach to the response 
response = make_flavor_pdf(response, flavor) 

return response 


This is a pretty straight-forward example, but if we have to leverage more mixins and deal with more 
custom logic, the simplicity of d j ango. vi ews . generi c. Vi ew makes it much easier than the more 
heavyweight views. In essence, we get all the advantages of function-based views combined with the 
object-oriented power that CBVs give us. 

10.7 Additional Resources 

>• docs.dj angoproj ect.com/en/1.11/topics/class-based-views/ 

>• docs.dj angoproj ect.com/en/1.11/topics/class-based-views/ 
generic-display/ 

>- docs.dj angoproj ect.com/en/1.11/topics/class-based-views/ 
generic-editing/ 

>• docs.dj angoproj ect.com/en/1.11/topics/class-based-views/mixins/ 

>- docs.dj angoproj ect.com/en/1.11/ref/class-based-views/ 

>■ The GCBV inspector at ccbv .co.uk 
>- python.org/download/releases/2.3/mro/ 

>- pydanny.com/tag/class-based-views.html 


PACKAGE TIP: Other Useful CBV Libraries 


>- django-extra-views Another great CBV library, django-extra-views covers the cases 
that django-braces does not. 

>- django-vaniUa-views A very interesting library that provides all the power of classic 
Django GCBVs in a vastly simplified, easier-to-use package. Works great in combina- 
tion with django-braces. 
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10.8 Summary 

This chapter covered: 

>- Using mixins with CBVs 

>- Which Django CBV should be used for which task 
>- General tips for CBV usage 
>- Connecting CBVs to forms 
>- Using the base dj ango. vi ews. generi c. Vi ew 


The next chapter explores common CBV/form patterns. Knowledge of these are helpful to have in 
your developer toolbox. 
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11 Common Patterns for Forms 


Django forms are powerfiil, flexible, extensible, and robust. For this reason, the Django admin and 
CBVs use them extensively. In fact, ali the major Django API frameworks use ModelForms or a 
similar implementation as part of their validation. 


Combining forms, models, and views allows us to get a lot of work done for little effort. The learning 
curve is worth it: once you learn to work fluently with these components, you’ll find that Django 
provides the ability to create an amazing amount of useful, stable functionality at an amazing pace. 


PACKAGE TIP: Useful Form-Related Packages 


>- django-floppyforms for rendering Django inputs in HTML5. 

>- django-crispy-forms for advanced form layout Controls. By default, forms are rendered 
with Twitter Bootstrap form elements and styles. This package plays well with django- 
floppyforms, so they are often used together. 

>■ django-forms-bootstrap is a simple tool for rendering Django forms using Twitter 
Bootstrap styles. This package plays well with django-floppyforms but conflicts with 
dj ango-crispy-forms. 


This chapter goes explicitly into one of the best parts of Django: forms, models, and CBVs working 
in concert. This chapter covers flve common form patterns that should be in every Django developer’s 
toolbox. 
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11.1 Pattern 1: Simple ModelForm With Default 
Validators 


The simplest data-changing form that we can make is a ModelForm using several default validators 
as-is, without modification. In fact, we already relied on default validators in Section 10.5.1: Views 
+ ModelForm Example. 


If you recaU, using ModelForms with CBVs to implement add/edit forms can be done in just a few 
lines of code: 



To summarize how we use default validation as-is here: 


>• FlavorCreateVi ew and FlavorUpdateVi ew are assigned Flavor as their model. 

>- Both views auto-generate a ModelForm based on the Flavor model. 

>■ Those ModelForms rely on the default field validation rules of the Flavor model. 

Yes, Django gives us a lot of great defaults for data validation, but in practice, the defaults are never 
enough. We recognize this, so as a first step, the next pattern will demonstrate how to create a custom 
field validator. 
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11.2 Pattern 2: Custom Form Field Validators in 
ModelForms 

What if we wanted to be certain that every use of the ti tle field across our project’s dessert apps 
started with the word ‘Tasty’? 



Figure 11.1: At Tasty Research, every fiavor must begin with “Tasty”. 


This is a string validation problem that can be solved with a simple custom field validator. 

In this pattern, we cover how to create custom single-field validators and demonstrate how to add 
them to both abstract models and forms. 

Imagine for the purpose of this example that we have a project with two different dessert-related mod¬ 
els: a Fiavor model for ice cream flavors, and a Mi Ikshake model for different types of millcshakes. 
Assume that both of our example models have ti tle fields. 

To validate all editable model tities, we start by creating a validators.py module: 
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In Django, a custom field validator is simply a callable (usually a function) that raises an error if the 
submitted argument doesnt pass its test. 

Of course, while our validate_tasty () validator function just does a simple string check for the 
sake of example, it’s good to keep in mind that form field validators can become quite complex in 
practice. 


TIP: Test Your Validators Carefully 


Since validators are critical in keeping corruption out of Django project databases, it’s espe- 
cially important to write detailed tests for them. 

These tests should include thoughtful edge case tests for every condition related to your val¬ 
idators’ custom logic. 


In order to use our validate_tasty () validator function across different dessert models, we’re 
going to first add it to an abstract model called TastyTi tleAbstractModel, whichwe plan to use 
across our project. 

Assuming that our Flavor and Milkshake models are in separate apps, it doesnt make sense to 
put our validator in one app or the other. Instead, we create a core/models.py module and place the 
TastyTitleAbstractModel there. 
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The last two lines of the above example code for core/models.py make TastyTi tleAbstractModel 
an abstract model, which is what we want. See Section 6.1.2: Be Careful With Model Inheritance. 

Let’s alter the ongmAflavors/models.py Flavor code to use TastyTitleAbstractModel as the 
parent class: 



This Works with the Flavor model, and it will work with any other tasty food-based model such 
as a Waf fleCone or Cake model. Any model that inherits from the TastyTitleAbstractModel 
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class will throw a validation error if anyone attempts to save a model with a title that doesn’t start 
with ‘Tasty’. 


Now, let’s explore a couple of questions that might be forming in your head: 


>■ What ifwe wanted to use validate_tasty 0 in just forms? 
>- What if we wanted to assign it to other fields besides the title? 


To support these behaviors, we need to create a custom FlavorForm that utilizes our custom field 
validator: 



A nice thing about both examples of validator usage in this pattern is that we havent had to change 
the vali date_tasty () code at all. Instead, we just import and use it in new places. 


Attaching the custom form to the views is our next step. The default behavior of Django model-based 
edit views is to auto-generate the ModelForm based on the views model attribute. We are going to 
override that default and pass in our custom FlavorForm. This occurs in th.e.flavors/views.py module, 
where we alter the create and update forms as demonstrated below: 
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The FlavorCreateVi ew and FlavorUpdateVi ewviews nowuse the new FlavorForm to validate 
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incoming data. 

Note that with these modifications, the Flavor model can either be identical to the one at the start 
of this chapter, or it can be an altered one that inherits from TastyTi tleAbstractModel. 

11.3 Pattern 3: Overriding the Clean Stage of Validation 

Let’s discuss some interesting validation use cases: 

>• Multi-field validation 

>- Validation involving existing data from the database that has already been validated 

Both of these are great scenarios for overriding the clean () and clean_<fi eld_name> () meth- 
ods with custom validation logic. 

After the default and custom field validators are run, Django provides a second stage and process 
for validating incoming data, this time via the clean () method and clean_<fi eld_name> () 
methods. You might wonder why Django provides more hooks for validation, so here are our two 
favorite arguments: 

O The clean () method is the place to validate two or more fields against each other, since it’s 
not specific to any one particular field. 

0 The clean validation stage is a better place to attach validation against persistent data. Since the 
data already has some validation, you wont waste as many database cycles on needless queries. 

Let’s explore this with another validation example. Perhaps we want to implement an ice cream 
ordering form, where users could specify the flavor desired, add toppings, and then come to our store 
and pick them up. 

Since we want to prevent users from ordering flavors that are out of stock, weTl put in a 
clean_slug () method. With our flavor validation, our form might look like: 
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class IceCreaniOrderForm(forms . Form) : 

"""Normally done with forms.ModelForm. But we use forms.Form here 
to demonstrate that these sorts of techniques work on every 
type of form. 

II II II 

slug = forms.ChoiceField(label= 'Flavor' ) 
toppings = forms.CharField() 

def _"init_ (self, *args, **kwargs) : 

super(IceCreamOrderForm, self) . _init_ (*args, 

**kwargs) 

# We dynamically set the choices here rather than 

# in the flavor field definition. Setting them in 

# the field definition means status updates won't 

# be reflected in the form without server restarts. 
self .fields [' slug ']. choices = [ 

(x.slug, x.title) for x in Flavor.objects.all() 

] 

# NOTE: We could filter by whether or not a flavor 

# has any scoops, but this is an example of 

# how to use clean_slug, not filter(). 

def clean_slug(self) : 

slug = self .cleaned_data [' slug' ] 

if Flavor.objects.get(slug=slug).scoops_remaining <= 0: 
msg = 'Sorry, we are out of that flavor.' 
raise forms.Vali dationError(msg) 
return slug 


For HTML-powered views, the clean_slug( ) method in our example, upon throwing an error, 
wiU attach a “Sorry, we are out of that flavor” message to the flavor HTML input fleld. This is a great 
shortcut for writing HTML forms! 

Now imagine if we get common customer complaints about orders with too much chocolate. Yes, it’s 
silly and quite impossible, but we’re just using ‘too much chocolate’ as a completely mythical example 
for the sake of making a point. 
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In any case, let’s use the clean() method to validate the flavor and toppings fields against each 
other. 



There we go, an implementation against the impossible condition of too much chocolate! 


TIP: Common Fields Used in Multi-Field Validation 


It is common practice for user account forms involved with email and password entry to force 
the user to enter the same data twice. Other things to check for against those fields include: 
>- Strength of the submitted password. 

>■ If the email model field isnt set to uni que=True, whether or not the email is unique. 
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Figure 11.2: Why would they do this to us? 


11.4 Pattern 4: Hacking Form Fields 
(2 CBVs, 2 Forms, 1 Model) 


This is where we start to get fancy. We’re going to cover a situation where two views/forms correspond 
to one model. We’U hack Django forms to produce a form with custom behavior. 


It’s not uncommon to have users create a record that contains a few empty fields which need additional 
data later. An example might be a list of Stores, where we want each store entered into the system as 
fast as possible, but want to add more data such as phone number and description later. Here’s our 
IceCreamStore model: 
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The default ModelForm for this modelforces the user to enter the ti tle and block_address field 
but allows the user to skip the phone and deseri pti on fields. Thats great for initial data entry, but 
as mentioned earlier, we want to have future updates of the data to require the phone and description 
fields. 

The way we implemented this in the past before we began to delve into their construction was to 
override the phone and description fields in the edit form. This resulted in heavily-duplicated code 
that looked like this: 


Example 11.10: Repeated HeavilyDuplicated Code 

# stores/forms.py 
from django import forms 

from .models import IceCreamStore 

class IceCreamStoreUpdateForm(forms.ModelForm): 

# Don't do this! Duplication of the model field! 
phone = forms.CharField(required=True) 

# Don't do this! Duplication of the model field! 
description = forms.TextField(required=True) 

class Meta: 
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model = IceCreamStore 


This form should look very familiar. Why is that? 

Well, we’re nearly copying the IceCreamStore model! 

This is just a simple example, but when dealing with a lot of fields on a model, the duplication becomes 
extremely challenging to manage. In fact, what tends to happen is copy-pasting of code from models 
right into forms, which is a gross violation of Don’t Repeat Yourself. 

Want to know how gross? Using the above approach, if we add a simple help_text attribute to 
the deseri ption field in the model, it will not show up in the template until we also modify the 
deseri ption field definition in the form. If that sounds confusing, thats because it is. 

A better way is to rely on a usefiil little detail that’s good to remember about Django forms: instanti- 
ated form objects store fields in a dict-like attribute called fi elds. 

Instead of copy-pasting field definitions from models to forms, we can simply modify existing at- 
tributes on specified fields in the_ i ni t _() method of the Model Form: 



157 








Chapter 11: Common Pattermfor Forms 


self .fields[ 'phone' ].required = True 
self .fields [' deseription']• required = True 


This improved approach allows us to stop copy-pasting code and instead focus on just the field- 
specific settings. 

An important point to remember is that when it comes down to it, Django forms are just Python 
classes. They get instantiated as objects, they can inherit from other classes, and they can act as 
superclasses. 

Therefore, we can rely on inheritance to trim the line count in our ice cream store forms: 
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WARNING: Use Meta.fields and Never Use Meta.exclude 


We use Heta . fi elds instead of Meta. exclude so that we know exactly what fields we 
are exposing. See Section 26.14: Don’t Use ModelForms.Meta.exclude. 


Finally, now we have what we need to define the corresponding CBVs. WeVe got our form classes, 
so let’s use them in the IceCreamStore create and update views: 



We now have two views and two forms that work with one model. 


11.5 Pattern 5: Reusable Search Mixin View 

In this example, we’re going to cover how to reuse a search form in two views that correspond to two 
different models. 

Assume that both models have a field called title (this pattern also demonstrates why naming 
standards in projects is a good thing). This example will demonstrate how a single CBV can be used 
to provide simple search functionality on both the Flavor and IceCreamStore models. 

We’ll start by creating a simple search mixin for our view: 
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The above code should look very familiar as we used it almost verbatim in the Forms + View example. 
Here’s how you make it work with both the Flavor and IceCreamStore views. First the flavor 
view: 



And weTl add it to the ice cream store views: 


Example 11.16: Adding TitleSearchMixin to IceCreamStoreListView 


# add to stores/views.py 

from django.views.generic import ListView 
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11.6: Summary 


from .models import Store 

from core.views import TitleSearchMixin 

class IceCreamStoreListView(TitleSearchMixin, ListView): 
model = Store 


As for the form? We just define it in HTML for each Li stVi ew: 



and 



Now we have the same mixin in both views. Mixins are a good way to reuse code, but using too many 
mixins in a single class makes for very hard-to-maintain code. As always, try to keep your code as 
simple as possible. 

11.6 Summary 

We began this chapter with the simplest form pattern, using a Model Form, CBV, and default val- 
idators. We iterated on that with an example of a custom validator. 

Next, we explored more complex validation. We covered an example overriding the clean methods. 
We also closely examined a scenario involving two views and their corresponding forms that were 
tied to a single model. 
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Finally, we covered an example of creating a reusable search mixin to add the same form to two 
different apps. 
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Form Fundamentals 



100% of Django projects should use Forms. 

95% of Django projects should use ModelForms. 
91% of ali Django projects use ModelForms. 

80% of ModelForms require trivial logic. 

20% of ModelForms require complicated logic. 


- pydanny made-up statistics™ 

Djangos forms are really powerful, and knowing how to use them anytime data is coming from 
outside your application is part of keeping your data clean. 

There are edge cases that can cause a bit of anguish. If you understand the structure of how forms are 
composed and how to caU them, most edge cases can be readily overcome. 

The most important thing to remember about Django forms is they should be used to validate all 
incoming data. 

12.1 Validate AU Incoming Data With Django Forms 

Djangos forms are a wonderful framework designed to validate Python dictionaries. While most 
of the time we use them to validate incoming FITTP requests containing POST, there is nothing 
limiting them to be used just in this manner. 

For example, let’s say we have a Django app that updates its model via CSV files fetched from another 
project. To handle this sort of thing, it’s not uncommon to see code like this (albeit in not as simplistic 
an example): 
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Example 12.1: How Not to Import CSV 

■import CSV 

from django.utils.six import StringlO 

from .models import Purchase 

def add_csv_purchases (rows): 

rows = StringlO.StringlO(rows) 
records_added = 0 

# Generate a dict per row, with the first CSV row being the keys 
for row in csv.DictReader(rows, delimiter=','): 

# DON'T DO THIS: Tossing unvalidated data into your model. 
Purchase.objects.create(**row) 
records_added += 1 
return records_added 


In fact, what you don’t see is that we’re not checking to see if seUers, stored as a string in the Pur¬ 
chase model, are actuallyvalid sellers. We could add validation code to our add_csv_purchases () 
function, but let’s face it, keeping complex validation code understandable as requirements and data 
changes over time is hard. 

A better approach is to validate the incoming data with a Django Form Hke so: 
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class Meta: 

model = Purchase 

def clean_seller (self ): 

seller = self . cleaned_data[ 'seller' ] 

try : 

Seller.obj ects . get(name=seller) 
except Seller . DoesNotExist: 

msg = '{0} does not exist in purchase #{1} format( 
seller, 

self . cleaned_data[ 'purchase_number' ] 

) 

raise forms.ValidationError(msg) 
return seller 

def add_csv_purchases(rows) : 

rows = StringlO . StringlO(rows) 

records_added = 0 
errors = [] 

# Generate a dict per row, with the first CSV row being the keys. 
for row in csv . DictReader(rows, delimiter= ',' ): 

# Bind the row data to the PurchaseForm. 
form = PurchaseForm(row) 

# Check to see if the row data is valid. 
if form . is_valid0 : 

# Row data is valid so save the record. 
form . save() 
records_added += 1 
else : 

errors . append(form . errors) 
return records_added, errors 


Whats really nice about this practice is that rather than cooking up our own validation system for 


165 







Chapter 12: Farm Fundamentah 


incoming data, we’re using the well-proven 


uaua uesuiig 



12.2 Use the POST Method in HTML Forms 

Every HTML form that alters data must submit its data via the POST method: 



The only exception you’ll ever see to using POST in forms is with search forms, which typically 
submit queries that don’t resuit in any alteration of data. Search forms that are idempotent should 
use the GET method. 


12.3 Always Use CSRF Protection With HTTP Forms That 
Modify Data 

Django comes with cross-site request forgery protection (CSRF) built in, and usage of it is introduced 
in Part 4 of the Django introductory tutorial. It’s easy to use, and Django even throws a friendly 
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warning during development when you forget to use it. This is a critical security issue, and here and 
in our security chapter we recommend always using Djangos CSRF protection capabilities. 

In our experience, the time when CSRF protection isn’t used is when creating machine-accessible 
APIs authenticated by proven libraries such as django-rest-f ramework-jwt. Tools Hke this, 
when combined with Django REST Framework do this for you. Since API requests should be 
signed/authenticated on a per-request basis, this means relying in HTTP cookies for authentication 
isn’t realistic. Therefore, CSRF isn’t always a problem when using these frameworks. 

If you are writing an API from scratch that accepts data changes, it’s a good idea to become familiar 

with Django’s CSRF documentation at 

docs.dj angoproj ect.com/en/1.11/ref/csrf/. 


TIP: HTML Search Forms 


Since HTML search forms don’t change data, they use the HTTP GET method and do not 
trigger Django’s CSRF protection. 


You should use Djangos CsrfVi ewMiddleware as blanket protection across your site rather than 
manually decorating views with csrf_protect. To ensure that CSRF works in Jinja2 templates, 
see Section 15.3: Considerations When Using Jinja2 With Django. 


12.3.1 Posting Data via AJAX 

You should use Djangos CSRF protection even when posting data via AJAX. Do not make your 
AJAX views CSRF-exempt. 

Instead, when posting via AJvLX, you’ll need to set an HTTP header called X-CSRFToken. 

The official Django documentation includes a snippet that shows how to set this header for 
only POST requests, in conjunction with jQuery 1.5.1 or higheFs cross-domain checking: docs. 
dj angoproj ect.com/en/1.ll/ref/csrf/#ajax 

See discuss this and more at Section 17.5: AJAX and the CSRF Token. 

Recommended reading: docs. dj angoproj ect. com/en/1.11/ref/csrf/ 
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12.4 Understand How to Add Django Form Instance 
Attributes 

Sometimes in the clean(), clean_FOO( ) or save () methods of a Django form, we need to have 
additional form instance attributes available. A sample case for this is having the request.user 
object available. Here is a simple taster-driven example. 

First, the form: 



See howwe set self .user before calling super(), and calls it from kwargs? Pointed out to us 
by Christopher Lambacher, this makes our form more robust, especially when using multiple inheri- 
tence. Now the view: 
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model = Taster 
form_class = TasterForm 
success_url = '/someplace/' 

def get_form_kwargs (self ): 

"""This method is what injects forms with keyword arguments. 

# grab the current set of form #kwargs 

kwargs = super (TasterUpdateView, self) .get_form_kwargs() 

# Update the kwargs with the user_id 
kwargs ['User' ] = self .request.user 
return kwargs 


PACKAGE TIP: django-braces’s ModelForm Mixins 


Inserting the request .user object into forms is so frequently done that django-braces can 
do it for us. Nevertheless, knowing how it works is useful for when what you need to add is 
not the request. user ohject. 

>- django-braces.readthedocs.io/en/latest/form.html# 
userformkwargsmixin 

>- django-braces.readthedocs.io/en/latest/form.html# 
userkwargmodelformmixin 


12.5 Know How Form Validation Works 

Form validation is one of those areas of Django where knowing the inner workings will drastically 
improve your code. Let’s take a moment to dig into form validation and cover some of the key points. 

When you caU f o rm. i s_vali d (), a lot of things happen behind the scenes. The following things 
occur according to this workflow: 

O If the form has bound data, form. is_valid () calls the form. full_clean () method. 
e form. full_clean () iterates through the form fields and each field validates itself: 

O Data Corning into the field is coerced into Python via the to_py thon () method or raises 
a ValidationError. 

O Data is validated against field-specific rules, including custom validators. Failure raises a 
ValidationError. 
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O If there are any custom clean_<fi eld> () methods in the form, they are called at this 
time. 

e form. full_clean 0 executes the form.clean() method. 

O Ifit’s a ModelForm instance, form._post_clean() does the foUowing: 

O Sets ModelForm data to the Model instance, regardless ofwhether form. is_valid () 
is True or False. 

O Calis the models clean () method. For reference, saving a model instance through the 
ORM does not call the models clean () method. 


If this seems complicated, just remember that it gets simpler in practice, and that all of this function- 
ality lets us reaUy understand whats going on with incoming data. The example in the next section 
should help to explain this further. 


D^RT.GR^SS, 

ANP lAOO 

^ ^ 


VALIDATION 

FAIL 

NOT £PIBL.e 


Figure 12.1: When ice cream validation fails. 


12.5.1 ModelForm Data Is Saved to the Form, Then the Model 
Instance 

We like to call this the WHAT?!? of form validation. At first glance, form data being set to the form 
instance might seem like a bug. But it’s not a bug. It’s intended behavior. 

In a ModelForm, form data is saved in two distinet steps: 

O First, form data is saved to the form instance. 

0 Later, form data is saved to the model instance. 
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Since ModelForms dont save to the model instance until they are activated by the form.saveO 
method, we can take advantage of this separation as a useful feature. 

For example, perhaps you need to catch the details of failed submissiori attempts for a form, saving 
both the user-supplied form data as well as the intended model instance changes. 

A simple, perhaps simplistic, way of capturing that data is as follows. First, we create a form failure 
history model in core/models.py as shown on the next page: 



Second, we add the following to the FlavorActi onMi xi n vaflavors/views.py. 



171 









Chapter 12: Farm Fundamentah 


def form_invalid (self , form): 

"""Save invalid form and model data for later reference.""" 
form_data = json.dumps(form.cleaned_data) 

# Seri alize the form.instance 

model_data = seri alizers.serialize (' json' , [form.instance]) 

# Strip away leading and ending bracket leaving only a dict 
model_data = model_data[1:-1] 

ModelFormFailureHistory.objects.create( 
form_data=form_data, 
model_data=model_data 

) 

return super (FlavorActionMixin, 

self) .form_invalid(form) 


If you recall, f orm_i nvali d () is called after failed validation of a form with bad data. When it is 
called here in this example, both the cleaned form data and the final data saved to the database are 
saved as a ModelFormFailureHistory record. 

12.6 AddErrorstoForms With Form.add_error() 

Shared with us by Michael Barr, we can streamline Form.clean() with the Form.add_error() 
method. 
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# Record errors that will be displayed later, 
msg = 'Coffee Ice Cream is not for Babies.' 
self . add_error( 'flavor' , msg) 
self . add_error( 'age' , msg) 

# Always return the full collectiori of cleaned data, 
return cleaned_data 


12.6.1 Other Useful Form Methods 

Here are other form validation methods worth exploring: 

>• docs.dj angoproj ect.com/en/1.11/ref/forms/api/#dj ango.forms.Form. 
errors.as_data 

>- docs.dj angoproj ect.com/en/1.11/ref/forms/api/#dj ango.forms.Form. 
errors.as_j son 

>- docs.dj angoproj ect.com/en/1.11/ref/forms/api/#dj ango.forms.Form.non 
field_errors 


12.7 Fields Without Pre-Made Widgets 

Two of the new dj ango. contri b. postgres fields, Ar rayFi eld and HStoreFi eld, don’t work 
well with existing Django HTML fields. They don’t come with corresponding widgets at all. Never- 
theless, you should stili be using forms with these fields. 

As in the previous section this topic is covered in Section 12.1: Validate All Incoming Data With 
Django Forms. 

12.8 Customizing W idgets 

One of our favorite features about Django 1.11 is trivial it is to override the HTML of Django 
widgets or even create custom widgets. This is a monumental change, a far cry from the days when 
most of us would do everything in our power to avoid these kinds of customizations. Heres our 
general advice: 
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>■ As always, keep it simple! Stay focused on presentation, nothing more. 

>- No widgets should ever change data. They are meant purely for display. 

>• FoUow the Django pattern and put aU custom widgets into modules called widgets.py. 


12.8.1 Overriding the HTML of Built-In Widgets 


This technique is useful for integrating tools like Bootstrap, Zurb, and other responsive front-end 
frameworks. On the downside, overriding the default templates in this manner means that every 
form element will use these alterations. To override these templates, you need to make the following 
changes to settings.py. 



Once that’s complete, create in your templates directory a directory at django/forms/templates and 
start overriding templates. If you want to know which templates can he overidden, you can look 
inside Django’s source code at github.com/django/django/tree/master/django/forms/ 
templates/django/forms/widgets 

More information: 


>- docs.djangoproject.com/en/1.11/ref/forms/renderers/ 

#overridi ng-buiIt-in-widget-templates 
>- docs.dj angoproj ect.com/en/1.11/ref/forms/renderers/#templatessetting 
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12.8.2 Creating New Custom Widgets 


If we want finer control of widgets, perhaps limiting the changes to certain data types, then it’s time 
to delve into creating custom widgets of our own. To do this: 

O Go to https://github.com/django/django/blob/master/django/forms/ 
widgets. py and select the widget closest to what you want. 

0 Extend the widget to behave as you would like. Keep changes to a minimum! 

Here’s an example: 



While this example is silly, it iUustrates what how to extend an existing widget to serve our purpose. 
Please note the foUowing: 

>• Ali the widget does is modify how the value is displayed. 

>■ The widget does not validate or modify the data coming back from the browser. That’s the job 
of forms and models respectively. 

>- We extended the absolute bare minimum of dj ango. forms. widgets.Textlnput to make it 
Work. 
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12.9 Additional Resources 

>- Author Daniel Roy Greenfelds blog series on forms expands upon concepts touched in this 
book pydanny.com/tag/forms. html 

>- BradMontgomery’sarticleonhowto create awidget forthe ArrayField . bradmontgomery. 

net/blog/2015/04/25/nice-arrayfield-widgets-choices-and-chosenj s// 

>• Rendering custom Django widgets docs. dj angoproj ect. com/en/1.11/ref/forms/ 
renderers/ 


12.10 Summary 

Once you dig into forms, keep yourself focused on clarity of code and testability. Forms are one of the 
primary validation tools in your Django project, an important defense against attacks and accidental 
data corruption. 

In the next chapter we’U dig into using templates. 
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Templates: Best Practices 



One of Djangos early design decisions was to limit the functionality of the template language. This 
heavily constrains what can be done with Django templates, which we often think is a very good 
thing since it forces us to keep business logic in the Python side of things. 

Think about it: the limitations of Django templates force us to put the most critical, complex and 
detailed parts of our project into . py files rather than into template files. Python happens to be one 
of the most ciear, concise, and elegant programming languages on the planet, so why would we want 
things any other way? 


TIP: Usingjinja2 With Django 


Since the release of 1.8, Django has natively supported Jinja2. It also provides an interface for 
including other template languages. We cover this topic in Chapter 15: Django Templates 
and Jinja2. 


13.1 Keep Templates Mostly in templates/ 

In our projects, we keep the majority of our templates in the main ‘templates/’ directory. We put 
subdirectories in ‘templates/’ to correspond to each of our apps, as shown here: 
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h- freezers/ 

I I- ("freezers" app templates in here) 


However, some tutorials advocate putting templates within a subdirectory of each app. We find that 
the extra nesting is a pain to deal with, as shown here: 



That said, some people like to do it the second way, and that’s alright. 

The exception to ali of this is when we workwith Django apps that are installed as pluggable packages. 
A Django package usually contains its own in-app ‘templates/’ directory. Then we override those 
templates anyway from our project’s main ‘templates/’ directory in order to add design and styling. 
We’ll explore this in Section 21.9: Releasing Your Own Django Packages 


13.2 Template Architecture Patterns 

We’ve found that for our purposes, simple 2-tier or 3-tier template architectures are ideal. The dif- 
ference in tiers is how many levels of template extending needs to occur before content in apps is 
displayed. See the examples below: 


13.2.1 2-Tier Template Architecture Example 

With a 2-tier template architecture, aU templates inherit from a single root base.html file. 
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Example 13.3: 2-Tier Template Architecture 


templates/ 

I- base.html 

I- dashboard.html # extends base.html 

I- profiles/ 

I I— profile_deta-il.html # extends base.html 

I I- profile_form.html # extends base.html 


This is best for sites with a consistent overall layout from app to app. 


13.2.2 3-Tier Template Architecture Example 

With a 3-tier template architecture: 

>- Each app has a base_<app_name>.html template. App-level base templates share a common 
parent base.html template. 

>- Templates within apps share a common parent base_<app_name>.html template. 

>- Any template at the same level as base.html inherits base.html. 


Example 13.4: 3-Tier Template Architecture 


templates/ 

base.html 

dashboard.html # extends base.html 
profiles/ 

base_profiles.html # extends base.html 
profile_deta-il.html # extends base_profiles.html 
profile_form.html # extends base_profiles.html 


The 3-tier architecture is best for websites where each section requires a distinctive layout. For exam¬ 
ple, a news site might have a local news section, a classified ads section, and an events section. Each 
of these sections requires its own custom layout. 

This is extremely useful when we want HTML to look or behave differently for a particular section 
of the site that groups functionality. 
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13.2.3 Flat Is Better Than Nested 






Gooi> NoT AS 

Figure 13.1: An excerpt from the Zen of Ice Cream. 


Complex template hierarchies make it exceedingly difRcult to debug, modify, and extend HTML 
pages and tie in CSS styles. When template block layouts become unnecessarily nested, you end up 
digging through file after file just to change, say, the width of a box. 

Giving your template blocks as shaUow an inheritance structure as possible wiU make your templates 
easier to work with and more maintainable. If youre working with a designer, your designer will 
thank you. 

That being said, theres a difference between excessively-complex template block hierarchies and tem¬ 
plates that use blocks wisely for code reuse. When you have large, multi-line chunks of the same or 
very similar code in separate templates, refactoring that code into reusable blocks wiU make your code 
more maintainable. 

The Zen of Python includes the aphorism ‘‘‘'Flat is better than nested” for good reason. Each level of 
nesting adds mental overhead. Keep that in mind when architecting your Django templates. 
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13.3: Limit Processing in Templates 


What you’ll see is the Zen of Python, an eloquently-expressed set of guiding principies for the 
design of the Python programming language. 


13.3 Limit Processing in Templates 


The less processing you try to do in your templates, the better. This is particularly a problem when it 
comes to queries and iteration performed in the template layer. 


Whenever you iterate over a queryset in a template, ask yourself the following questions: 

O How large is the queryset? Looping over gigantic querysets in your templates is almost always 
a bad idea. 

0 How large are the objects being retrieved? Are all the fields needed in this template? 

© During each iteration of the loop, how much processing occurs? 


If any warning bells go off in your head, then theres probably a better way to rewrite your template 
code. 


WARNING: Why Not Just Cache? 


Sometimes you can just cache away your template inefficiencies. That’s fine, but before you 
cache, you should first try to attack the root of the problem. 

You can save yourself a lot of work by mentally tracing through your template code, doing 
some quick run time analysis, and refactoring. 


Let’s now explore some examples of template code that can be rewritten more efliciently. 


Suspend your disbelief for a moment and pretend that the nutty duo behind Two Scoops ran a 30- 
second commercial during the Super Bowl. “Free pints of ice cream for the first million developers 
who request them! AU you have to do is fili out a form to get a voucher redeemable in Stores!” 
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SuPtR 

Bowl 

LWIII 




SCOOPS 

IWTeRNATlO^^^L 


U A L TT I M E 


OFFeR 


Fl?££ ICE CREAM FOR THE FIRST 


1 ^ 000,000 

PEVtLOPERS 



Figure 13.2: Two Scoops, official halftime sponsor of the Super Bowl. 


Naturally, we have a “vouchers” app to track the names and email addresses of everyone who requested 
a free pint voucher. Heres what the model for this app looks like: 
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redeemed = models.DateTimeField (null=True , default=None) 
objects = VoucherManager0 

This model will be used in the following examples to illustrate a few “gotchas” that you should avoid. 


13.3.1 Gotcha 1: Aggregation in Templates 

Since we have birth date Information, it would be interesting to display a rough breakdown by age 
range of voucher requests and redemptions. 

A very bad way to implement this would be to do all the processing at the template level. To be more 
specific in the context of this example: 

>- Don’t iterate over the entire voucher list in your template’s JavaScript section, using JavaScript 
variables to hold age range counts. 

>• Don’t use the add template filter to sum up the voucher counts. 

Those implementations are ways of getting around Djangos limitations of logic in templates, but 
they’ll slow down your pages drastically. 

The better way is to move this processing out of your template and into your Python code. Sticking 
to our minimal approach of using templates only to display data that has already been processed, our 
template loolcs like this: 
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</tr> 

</thead> 

<tbody> 

{% for age_bracket in age_brackets ?o} 
<tr> 

<td>{{ age_bracket.title }}</td> 
<td>{{ age_bracket.count }}</td> 
</tr> 

{% endfor 96 } 

</tbody> 

</table> 

{96 endblock content 96 } 


In this example, we can do the processing with a model manager, using the Django ORM’s aggrega- 
tion methods and the handy dateutil library described in Appendix A: Packages Mentioned In This 
Book: 
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{'title': '18+', 'count': count} 

) 

return age_brackets 


This method would be called from a view, and the results would be passed to the template as a context 
variable. 


13.3.2 Gotcha 2: Filtering With Conditionals in Templates 


Suppose we want to display a list of ali the Greenfelds and the Roys who requested free pint vouchers, 
so that we could invite them to our Family reunion. We want to filter our records on the name field. 
A very bad way to implement this would be with giant loops and if statements at the template level. 


Example 13.8: Disastrous Method of Filtering Data 

<h2>Greenfelds Who Want Ice Cream</h2> 

<ul> 

{% for voucher in voucher_list %} 

{# Don't do this: conditional filtering in templates #} 
{?o if 'greenfeld' in voucher. name. lower %} 

<li>{{ voucher.name }}</li> 
endif 96 } 

{96 endfor 95} 

</ul> 

<h2>Roys Who Want Ice Cream</h2> 

<ul> 

{96 for voucher in voucher_list 96 } 

{# Don't do this: conditional filtering in templates #} 
{95 if 'roy' in voucher. name. lower 95} 

<li>{{ voucher.name }}</li> 

{95 endi f 95} 

{95 endfor 95} 

</ul> 
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In this bad snippet, we’re looping and checking for various “if” conditions. Thats filtering a poten- 
tially gigantic list of records in templates, which is not designed for this kind of work, and wiU cause 
performance bottlenecks. On the other hand, databases like PostgreSQL and MySQL are great at 
filtering records, so this should be done at the database layer. The Django ORM can help us with 
this as demonstrated in the next example. 



Then to call the results, we use the following, simpler template: 
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{% endfor %} 

</ul> 


It’s easy to speed up this template by moving the filtering to a view. With this change, we now simply 
use the template to display the already-filtered data. The above template now follows our preferred 
minimalist approach. 


13.3.3 Gotcha 3: Complex Implied Queries in Templates 


Despite the limitations on logic allowed in Django templates, it’s all too easy to find ourselves caUing 
unnecessary queries repeatedly in a view. For example, if we list users of our site and aU their flavors 
this way: 


Example 13.11: Template Code Generating Extra Queries 

{# list generated via User.objects.all() #} 

<hl>Ice Cream Fans and their favorite flavors.</hl> 

<ul> 

{% for user in user_list %} 

<li> 

{{ user.name }}: 

{# DON'T DO THIS: Generated implicit query per user #} 
{{ user.flavor.title }} 

{# DON'T DO THIS: Second implicit query per user!!! #} 
{{ user.flavor.scoops_remaining }} 

</li> 

{% endfor ?o} 

</ul> 


Then caUing each user generales a second query. While that might not seem like much, we are certain 
that if we had enough users and made this mistake frequently enough, our site would have a lot of 
trouble. 

One quick correction is to use the Django ORM’s select_related () method: 
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One more thing: If youVe embraced using model methods, the same applies. Be cautious putting too 
much query logic in the model methods caUed from templates. 


13.3.4 Gotcha 4: Hidden CPU Load in Templates 

Watch out for innocent-looking calls in templates that resuit in intensive CPU processing. vUthough 
a template might look simple and contain very little code, a single line could he invoking an object 
method that does a lot of processing. 


REGiUiRes iqpoo% 

CHEW1N6 rHAN OTH€l? FlAVO/?S. 

Figure 13.3: Bubble gum ice cream looks easy to eat but requires a lot of processing. 




Common examples are template tags that manipulate images, such as the template tags provided by 
libraries like sorl-thumbnail. In many cases tools like this work great, but weVe had some issues. 
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Specifically, the manipulation and the saving of image data to file Systems (often across networks) 
inside a template means there is a choke point within templates. 

This is why projects that handle a lot of image or data processing increase the performance of their 
site by taking the image processing out of templates and into views, models, helper methods, or 
asynchronous messages queues like Celery or Django Channels. 


13.3.5 Gotcha 5: Hidden REST API Calis in Templates 

You saw in the previous gotcha how easy it is to introduce template loading delays by accessing object 
method calls. This is true not just with high-load methods, but also with methods that contain REST 
API calls. A good example is querying an unfortunately slow maps API hosted by a third-party 
Service that your project absolutely requires. Don’t do this in the template code by calling a method 
attached to an object passed into the view’s context. 

Where should actual REST API consumption occur? We recommend doing this in: 

>- JavaScript code so after your project serves out its content, the clients browser handles the 
Work. This way you can entertain or distract the client while they wait for data to load. 

>- The view’s Python code where slow processes might be handled in a variety of ways including 
message queues, additional threads, multiprocesses, or more. 


13.4 Don’t Bother Making Your Generated HTML Pretty 

Bluntly put, no one cares if the HTML generated by your Django project is attractive. In fact, if 
someone were to look at your rendered HTML, they’d do so through the lens of a browser inspector, 
which would realign the HTML spacing anyway. Therefore, if you shuffle up the code in your Django 
templates to render pretty HTML, you are wasting time obfuscating your code for an audience of 
yourself 

And yet, we ve seen code like the following. This evil code snippet generates nicely formatted HTML 
but itself is an illegible, unmaintainable template mess: 
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Example 13.13: Obfuscating Template Code to Produce Pretty HTML Code 

{% comment ?o}Don't do this! This code bunches everything 
together to generate pretty HTML, 
endcomment %y 

{% if list_type=='unordered ' 5o}<ul>{?o else ?o}<ol>{?o endif ?o}{°o for 
syrup in syrup_Tist ?o}<li class="{{ syrup. temperature_type | roomtemp 
}}"><a href="{?o uri ' syrup_detai 1' syrup. slug ?o}">{°» syrup. title %} 
</a></li>{9o endfor %}{% if list_type== 'unordered ' 9o}</ul>{5o else %} 
</ol>{5o endif %} 


A better way of writing the above snippet is to use indentation and one operation per line to create 
a readable, maintainable template: 
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Are you worried about the volume of whitespace generated? Don’t be. First of ali, experienced de- 
velopers favor readability of code over obfuscation for the sake of optimization. Second, there are 
compressiori and minification tools that can help more than anything you can do manually here. See 
Chapter 24: Finding and Reducing Bottlenecks for more details. 


13.5 Exploring Template Inheritance 

Let’s begin with a simple base.html file that we’ll inherit from another template: 



The base.html file contains the foUowing features: 

>- A tltle blockcontaining “Two Scoops of Django”. 

>- A stylesheets block containing a link to zproject.css file used across our site. 
>• A content block containing “<hl>Two Scoops</hl>”. 
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Our example relies on just three template tags, which are summarized below: 


Template Tag 

Purpose 


load 96} 

Loads the staticfiles built-in template tag library 



Since base.html is a parent template, these define which child blocks can be 


block 96} 

filled in by child templates. We place links and Scripts inside them so we can 
override if necessary. 

J ‘ 0 ;' 

\/0 

stati c 96} 

Resolves the named static media argument to the static media server. 


Table 13.1: Template Tags in base.html 


To demonstrate base.html in use, weT have a simple about.html inherit the foUowing from it: 
>- A custom title. 

>- The original stylesheet and an additional stylesheet. 

>- The original header, a sub header, and paragraph content. 

>■ The use of child blocks. 

>- The use of the { { block. super }} template variable. 
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When we render this template in a view, it generates the following HTML: 


Example 13.17: Rendered HTML 


<html> 

<head> 

<title> 

About Audrey and Daniel 
</title> 

<l"ink rel="stylesheet" type="text/css" 
href="/static/css/proj ect.css"> 
<link rel="stylesheet" type="text/css" 
href="/static/css/about.css"> 

</head> 

<body> 

<div class="content"> 

<hl>Two Scoops</hl> 

<h2>About Audrey and Daniel</h2> 
<p>They enjoy eating ice cream</p> 

</div> 

</body> 

</html> 


Notice how the rendered HTML has our custom title, the additional stylesheet link, and more mate- 
rial in the body? We’ll use the table below to review the template tags and variables in the about.html 
template. 


Template Object Purpose 


Informs Diango that about.html is inheriting or extending from 
{96 extends 96 } ^ b b 

base.html 

Since about.html is a child template, block overrides the content 
{96 block 96} provided by base.html. This means our title will render as 

<title>Audrey and Daniel</title>. 

When placed in a child templates block, it ensures that the parent’s 
{{ block. super }} content is also included in the block. In the content blockof the 
about.html template, this wiU render <hl>Two Scoops</hl>. 
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Template Object Purpose 


Table 13.2: Template Objects in about.html 


Note that the {96 block 96} tag is used differently in about.html than in base.html, serving to over- 
ride content. In blocks where we want to preserve the base.html content, we use {{ block. super 
}} variable to display the content from the parent block. This brings us to the next topic, {{ 
block.super }}. 

13.6 block.super Gives the Power of Control 

Let’s imagine that we have a template which inherits everything from the base.html but replaces the 
projects link to project.css file with a link to dashboard.css. This use case might occur when you 
have a project with one design for normal users, and a dashboard with a different design for staff. 

If we arent using {{ block.super }}, this often involves writing a whole new base file, often 
named something like base_dashboard.html. For better or for worse, we now have two template ar- 
chitectures to maintain. 

If we are using {{ block. super }}, we dont need a second (or third or fourth) base template. 
Assuming all templates extend from base.html we use {{ block. super }} to assume control of 
our templates. Here are three examples: 

Template using ho'^ project.css and a custom link: 



Dashboard template that excludes the. project.css link: 
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Template just linking the.project.css file: 



These three examples demonstrate the amount of control that {{ block. super }} provides. The 
variable serves a good way to reduce template complexity, but can take a little bit of effort to fully 
comprehend. 


TIP: block,super Is Similar but Not the Same as super() 


For those coming from an object oriented programming background, it might help to think 
of the behavior of the {{ block.super }} variable to be like a very limited version of the 
Python built-in function, super (). In essence, the {{ block. super }} variable and the 
super () function both provide access to the parent. 

Just remember that they arent the same. For example, the {{ block.super }} variable 
doesht accept arguments. It’s just a nice mnemonic that some developers might find usefiil. 
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13.7 Useful Things to Consider 

The following are a series of smaller things we keep in mind during template development. 


13.7.1 Avoid Coupling Styles Too Tightly to Python Code 

Aim to control the styling of all rendered templates entirely via CSS and JS. 

Use CSS for styling whenever possible. Never hardcode things like menu bar widths and color choices 
into your Python code. Avoid even putting that type of styling into your Django templates. 

Here are some tips: 

>- If you have magic constants in your Python code that are entirely related to visual design layout, 
you should probably move them to a CSS file. 

>- The same applies to JavaScript. 


13.7.2 Common Conventions 

Here are some naming and style conventions that we recommend: 

>- We prefer underscores over dashes in template names, block names, and other names in tem¬ 
plates. Most Django users seem to follow this convention. Why? Well, because underscores 
are aUowed in names of Python objects but dashes are forbidden. 

>- We rely on ciear, intuitive names forblocks. {96 block javascript %} is good. 

>- We include the name of the block tag in the endblock. Never write just {96 endblock ^o}) 
include the whole {% endblock javascript 96}. 

> Templates called by other templates are prefixed with This applies to templates called via 
{96 1 nclude 96} or custom template tags. It does not apply to templates inheritance Controls 
such as {% extends 96} or {% block 96}. 


196 



13.7: Useful Things to Consider 


13.7.3 Use Implicit and Named Explicit Context Objects Properly 

When you use generic display CBVs, you have the option of using the generic {{ ob j ect_li st 
}} and {{ object }} inyour template. Another option is to use the ones that are named afteryour 
model. 


For example, if you have a Topping model, you can use {{ toppi ng_li st }} and {{ toppi ng 
}} inyour templates, instead of {{ object_list }}and{{ object }}. This means both of the 
following template examples wiU work: 


Example 13.21: Implicit and Explicit Context Objects 

{# 

templates/toppi 

ngs/topping_list 

;.html #} 

{# 

Using implicit 

names, good for 

code reuse #} 

<ol> 




for object in object_list 



<li>{{ object 

}} </li> 



endfor %} 



</ol> 



{# 

Using explicit 

names, good for 

object speci fic code #} 

<ol> 




for topping in 

topping_list %} 



<li>{{ topping 

}} </li> 


{.% 

endfor %} 



j </0l> 




13.7.4 Use URL Names Instead of Hardcoded Paths 

A common developer mistake is to hardcode URLs in templates like this: 


Example 13.22: Hardcoded URL in Template. Sad! 
<a href="/flavors/"> 


The problem with this is that if the URL patterns of the site need to change, all the URLs across the 
site need to be addressed. This impacts HTML, JavaScript, and even RLSTful APIs. 
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Instead, we use the {% uri %} tag and references the names in our URLConf files: 



13.7.5 Debugging Complex Templates 

A trick recommended by Lennart Regebro is that when templates are complex and it becomes difRcult 
to determine where a variable is failing, you can force more verbose errors through the use of the 
string_i f_invalid option in OPTIONS ofyour TEMPLATES setting: 



13.8 Error Page Templates 

Even the most tested and analyzed site will have a few problems now and then, and that’s okay The 
problem lies in how you handle those errors. The last thing that you want to do is show an ugly 
response or a blank web server page back to the end user. 

It’s Standard practice to create at least 404.html and 500.html templates. See the GitHub HTML 
Styleguide link at the end of this section for other types of error pages that you may want to consider. 

We suggest serving your error pages from a static file server (e.g. Nginx or Apache) as entirely self- 
contained static HTML files. That way, if your entire Django site goes down but your static file server 
is stili up, then your error pages can stiU be served. 
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13.9: Follow a Minimalist Approach 


If youre on a PaaS, check the documentation on error pages. For example, Heroku allows users to 
upload a custom static HTML page to be used for 500 errors. 


WARNING: Resist the Temptation to Overcomplicate Your 
Error Pages 


Interesting or amusing error pages can be a draw to your site, but don’t get carried away. It’s 
embarrassing when your 404 page has a broken layout or your 500 page can’t load the CSS 
and JavaScript. Worse yet is dynamic 500 error pages that break in the event of a database 
failure. 


GitHubs 404 and 500 error pages are great examples of fancy but entirely static, self-contained error 
pages: 

>- github.com/404 

>- github.com/500 

View the source of either of them and you’ll notice that: 

>- Ali CSS styles are inline in the head of the same HTML page, eliminating the need for a 
separate stylesheet. 

>■ All images are entirely contained as data within the HTML page. There are no <img> links 
to external URLs. 

>■ All JavaScript needed for the page is contained within the HTML page. There are no external 
links to JavaScript assets. 

For more information, see the Github HTML Styleguide: 

>• github.com/styleguide 


13.9 Follow a Minimalist Approach 

We recommend taking a minimalist approach to your template code. Treat the so-called limitations 
of Django templates as a blessing in disguise. Use those constraints as inspiration to find simple, 
elegant ways to put more of your business logic into Python code rather than into templates. 
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Taking a minimalist approach to templates also makes it much easier to adapt your Django apps to 
changing format types. When your templates are bulky and full of nested looping, complex condition- 
als, and data processing, it becomes harder to reuse business logic code in templates, not to mention 
impossible to use the same business logic in template-less views such as API views. Structuring your 
Django apps for code reuse is especially important as we move forward into the era of increased API 
development, since APIs and web pages often need to expose identical data with different formatting. 

To this day, HTML remains a Standard expression of content, providing the practices and patterns 
for this chapter. 

13.10 Summary 

In this chapter, we covered the following: 

>- Template inheritance, including the use of {{ block. super }}. 

>- Writing legible, maintainable templates. 

>- Easy methods to optimize template performance. 

>- Issues with limitations of template processing. 

>- Error page templates. 

>- Many other helpful little details about templates. 

In the next chapter we’ll examine template tags and filters. 
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Template Tags and Filters 



Django provides dozens of default template tags and filters, all of which share the following common 
traits: 

>• All of the defaults have ciear, obvious names. 

>• All of the defaults do just one thing. 

>■ None of the defaults alter any sort of persistent data. 

These traits serve as very good best practices when you have to write your own template tags. Let’s 
now dive a bit deeper into practices and recommendations when writing custom filters and template 
tags. 


14.1 Filters Are Functions 

Filters are functions that accept just one or two arguments, and that doht give developers the ability 
to add behavior Controls in Django templates. 

We feel that this simplicity makes filters less prone to abuse, since they are essentially just functions 
with decorators that make Python usable inside of Django templates. This means that they can be 
called as normal functions (although we prefer to have our filters caU functions imported from utility 
modules). 

In fact, a quick scan of the source code of Djangos default filters at /gi t. io/vyzxO shows that the 
slugi fy () template filter simply calls the django. uti Is . text. slugi fy function. 
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Figure 14.1: This filter transforms 1-2 flavors of ice cream into vanilla, outputting to a cone. 


14,1,1 Filters Are Easy to Test 

Testing a filter is just a matter of testing a function, which we cover in Chapter 22: Testing Stinks 
and Is a Waste of Money!. 


14,1,2 Filters and Code Reuse 

As can be seen in the Django 1.11 source code defaulifilter.py github.com/django/ 
dj ango/blob/stable/1.11.x/dj ango/template/defaultfiIters.py, most 

of the filter logic is imported from other libraries. For example, it’s not neces- 
sary to import dj ango. template. defaultfi Iters. slugi fy. Instead we can use 
dj ango. uti Is. text. slugi fy. While it might seem to be perfectly acceptable to import 
the filter, it adds a level of code abstraction that can make debugging a problem a little more 
difiicult. 

Since filters are just functions, we advocate that anything but the simplest logic for them be moved 
to more reusable utiHty functions, perhaps stored in a utils.py module. Doing this makes it easier to 
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14.2: Custom Template Tags 


introspect code bases and test, and can mean dramatically fewer imports. Over time core Django has 
followed this pattern more and more. 


14.1,3 When to Write Filters 

Filters are good for modifying the presentation of data, and they can be readily reused in REST APIs 
and other output formats. Being constrained to two arguments limits the functionality so it’s harder 
(but not impossible) to make them unbearably complex. 

14.2 Custom Template Tags 

“Please stop writing so many template tags. They are a pain to debugP 

- Audrey Roy Greenfeld, while debugging Daniel Roy Greenfeld’s code. 

While template tags are great tools when developers have the discipline to keep them in check, in 
practice they tend to get abused. This section covers the problems that you run into when you put 
too much of your logic into template tags and filters. 


14,2,1 Template Tags Are Harder to Debug 


Template tags of any complexity can be chaUenging to debug. When they include opening and closing 
elements, they become even harder to handle. WeVe found liberal use of log statements and tests are 
very helpful when they become hard to inspect and correct. 


14,2,2 Template Tags Make Code Reuse Harder 

It can be diflicult to consistently apply the same effect as a template tag on alternative output formats 
used by REST APIs, RSS feeds, or in PDF/CSV generation. If you do need to generate alternate 
formats, it’s worth considering putting aU logic for template tags into utils.py, for easy access from 
other views. 


203 



Chapter 14: Template Tags and Filters 


14.2.3 The Performance Cost of Template Tags 

Template tags can have a significant performance cost, especially when they load other templates. 
While templates run much faster than they did in pre-1.8 versions of Django, it’s easy to lose those 
performance benefits if you don’t have a deep understanding of how templates are loaded in Django. 

If your custom template tags are loading a lot of templates, you might want to consider caching the 
loaded templates. See docs.djangoproject.com/en/1.11/ref/templates/api/#django. 
template. loaders. cached. Loader for more details. 


14.2.4 When to Write Template Tags 

These days, we’re very cautious about adding new template tags. We consider two things before 
writing them: 

>- Anything that causes a read/write of data might be better placed in a model or object method. 
>- Since we implement a consistent naming Standard across our projects, we can add an abstract 
base class model to our core.models module. Can a method or property in our project’s 
abstract base class model do the same work as a custom template tag? 

When should you write new template tags? We recommend writing them in situations where they are 
only responsible for rendering of HTML. For example, projects with very complex HTML layouts 
with many different models or data types might use them to create a more fiexible, understandable 
template architecture. 


PACKAGE TIP: We Do Use Custom Template Tags 


It sounds like we stay away from custom template tags, but thats not the case. We’re just cau¬ 
tious. Interestingly enough, Daniel has been involved with at least three prominent libraries 
that make extensive use of template tags. 

>- django-crispy-forms 
>- django-wysiwyg 

>■ django-uni-form (deprecated, use django-crispy-forms instead) 
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14.3 Naming Your Template Tag Libraries 


The convention we follow is <app_name>_tags.py. Using the twoscoops example, we would have files 
named thus: 


>- Jlavors_tags.py 

> blog_tags.py 

>■ events_tags.py 

> tickets_tags.py 


This makes determining the source of a template tag library trivial to discover. 


TIP: Doii’t Name Your Template Tag Libraries With the Same Name as 
Your App 


For example, naming the events app’s templatetag library events.py is problematic. 

It used to be that doing so would cause ali sorts of problems because of the way that Django 
loaded template tags. While thats been fixed, the convention of adding a _tags.py suflix to 
template tag libraries has remained. This means it’s easy for everyone to find template tag 
libraries. 


WARNING: Don’t Use Your IDE’s Features as an Excuse to 
Obfiiscate Your Code 


Do not rely on your text editor or IDE’s powers of introspection to determine the name of 
your template tag library. 


14.4 Loading Your Template Tag Modules 


In your template, right after {9o extends "base.html" %} (or anyother parent template besides 
base.html) is where you load your template tags: 
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Simplicity itself! Explicit loading of functionality! Hooray! 


14,4,1 Watch Out for This Anti-Pattern 

Unfortunately, there is an anti-pattern that will drive you mad with fury each and every time you 
encounter it: 


Example 14.2: Implicit Loading of Template Tag Libraries 

# settings/base.py 
TEMPLATES = [ 

'BACKEND': 'dj ango.template.backends.dj ango.Dj angoTemplates', 
'OPTIONS': { 

# Don't do this! 

# It's an evil anti-pattern! 

'buiItins ' : ['flavors.templatetags.flavors_tags'], 

}, 

] 


The anti-pattern replaces the explicit load method described above with an implicit behavior which 
supposedly fixes a “Dont Repeat Yourself” (DRY) issue. However, any DRY “improvements” it 
creates are destroyed by the following: 

>■ It will add some overhead due to the fact this literally loads the template tag library into each 
and every template loaded by dj ango, template.Template. This means every inherited 
template, template {96 include 96}, inclusion_tag, and more wiU be impacted. While 
we have cautioned against premature optimization, we are also not in favor of adding this 
much unneeded extra computational work into our code when better alternatives exist. 

>- Because the template tag library is implicitly loaded, it immensely adds to the difRculty in 
introspection and debugging. Per the Zenof Python, “Explicit is better than Implicit.” 
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14.5: Summary 


Fortunately, this is obscure because beginning Django developers don’t know enough to make this 
mistake and experienced Django developers get annoyed when they have to deal with it. 

14.5 Summary 

It is our contention that template tags and filters should concern themselves only with the manipu- 
lation of presentable data. So long as we remember this when we write or use them, our projects run 
faster and are easier to maintain. 

In the next chapter we explore the use of Django templates and Jinja2. 
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Django Templates and Jinja2 



With Django 1.8 came support for multiple template engines. As of now, the only available built-in 
backends for the Django template system are the Django template language (DTL) andjinja2. 

15.1 What’s the Syntactical DifFerence? 

At the syntax level, DTL and Jinja2 are very similar. In fact, Jinja2 was inspired by DTL. Here are 


the most significant syntax differences: 

1 Subject 

DTL 


Jinja2 

Method Calis 

{{ user.get_favorites 

» 

{{ user.get_favorites0 




}} 

Filter Arguments 

{{ toppingsljoin: ' , ' 

» 

{{ toppings1join(', ') }} 

Loop Empty Argument 

{% empty 96} 


{96 else 96} 

Loop Variable 

{{ forloop }} 


{{ loop }} 

Cycle 

{96 cycle ' odd ' 'even' 

96} 

{{ loop.cycle('odd', 


'even') }} 

Table 15.1: DTLvsJinja2 Syntax Differences 
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15.2 Should I Switch? 

First off, when using Django we don’t have to choose between DTL or Jinja2. We can set set- 
ti ngs . TEMPLATES to use DTL for some template directories and Jinja2 for others. Ifwe have a lot 
of templates in our codebase, we can hold onto our existing templates and leverage the benefits of 
Jinja2 where we need it. This aUows for the best of both worlds: Access to the vast Django ecosystem 
of third-party packages and the features of alternatives to DTL. 

In short, we can use multiple template languages together harmoniously. 

For example, most of a site can be rendered using DTL, with the larger pages rendered content done 
with Jinja2. A good example of this behavior are the djangopackages.org/grids. Because of 
their size and complexity, in the near future these pages may be refactored to be powered byjinja2 
rather than DTL. 


15.2.1 Advantages of DTL 

Here are reasons to use the Django template language: 

>■ It’s batteries-included with all the functionality clearly documented within the Django docs. 
The oflicial Django documentation on DTL is very extensive and easy to follow. The template 
code examples in the Django docs use DTL. 

>- The DTL+Django combination is much more tried and mature than the Jinja2+Django com- 
bination. 

>- Most third-party Django packages use DTL. Converting them to Jinja2 is extra work. 

>■ Converting a large codebase from DTL to Jinja2 is a lot of work. 

15.2.2 Advantages of Jinja2 

Here are reasons to use Jinja2: 

>- Can be used independently of Django. 

>- As Jinja2’s syntax is closer to Pythons syntax, many find it more intuitive. 

>■ Jinja2 is generally more explicit, e.g. function calls in the template use parentheses. 
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15.3: Consideratioris When Usingjinja2 With Django 


>■ Jinja2 has less arbitrary restrictions on logic, e.g. you can pass unlimited arguments to a filter 
with Jinja2 vs. only 1 argument with DTL. 

>■ According to the benchmarks onbne and our own experiments, Jinja2 is generally faster. That 
said, templates are always a much smaller performance bottleneck than database optimization. 
See Chapter 24: Finding and Reducing Bottlenecks. 


15.2.3 Which One Wins? 

It depends on your situation: 

>- New users should always stick with DTL. 

>- Existing projects with large codebases will want to stick with DTL except for those few pages 
that need performance improvements. 

>- Experienced Djangonauts should try both, weigh the benefits of DTL and Jinja, and make 
their own decision. 


TIP: Choose a Primary Template Language 


While we can mix multiple template languages across a project, doing so risks adding dra- 
matically to the mental overload of a project. To mitigate this risk, choose a single, primary 
template language. 


15.3 Considerations When Using Jinja2 With Django 

Here are some things to keep in mind when using Jinja2 templates with Django: 


15.3.1 CSRF andjinja2 

Jinja2 accesses Djangos CSRF mechanism dilferently than DTL. To incorporate CSRF into Jinja2 
templates, when rendering forms make certain to include the necessary HTML: 
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15.3.2 Using Template Tags in Jinja2 Templates 

At this time using Django-style Template Tags isn’t possible in Jinja2. If we need the functionality 
of a particular template tag, depending on what we’re trying to do we convert it using one of these 
techniques: 

>- Convert the functionality into a function. 

>- Create a Jinja2 Extension. See jinja.pocoo.org/docs/dev/extensions/ 
#module-j 1 nj a2.ext 


15.3.3 Using Django-Style Template FUters in Jinja2 Templates 

One thing weVe grown used to having around in DTL is Djangos default template filters. Fortu- 
nately, as Django filters are just functions (see Section 14.1: Filters Are Functions), we can easily 
specify a custom Jinja2 environment that includes the template filters: 
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' uri' : reverse, 

'dj' : defaultfilters 

}) 

return env 


Here is a an example of using Django templa te filters as functions in a Jinja2 template: 



If you want a less global approach, we can use a technique explored in Section 10.4.3: Performing 
Custom Actions on Views With Invalid Forms. Here we create a mixin for attaching the Django 
template filters as an attribute on views: 



If a view inherits from our core. mi xi ns. Dj Fi IterMi xi n class, in its Jinja2 template we can do 
the following: 
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TIP: Avoid Using Context Processors With Jiiija2 


The Django documentation recommends against using context processors with Jinja2. See the 
warning box at docs.djangoproject.com/en/1.ll/topics/templates/#django. 
template. backends. j i nj a2.nj a2. What they recommend instead is passing a func- 
tion to the template that can be called as needed. This can be done on a per-view basis or by 
injecting the callable function as described in this subsection. 


15.3.4 The Jinja2 Environment Object Should Be Considered Static 

Inexample 15.1 we demonstrate the use ofthe core component ofJinja2, the j i nj a2. Envi ronment 
class. This object is where Jinja2 shares configuration, filters, tests, globals, and more. When the first 
template in your project is loaded, Jinja2 instantiates this class as what is essentially a static object. 

Example: 
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import random 

def envi ronment(**options) : 

env = Environment(**options) 
env.globaIs.update({ 

# Runs only on the first template load! The three displays below 

# will all present the same number. 

# {{ random_once }} {{ random_once }} {{ random_once }} 

'random_once' : random.randint(1, 5) 

# Can be called repeated as a function in templates. Each call 

# returns a random number: 

# {{ randomO }} {{ random() }} {{ random() }} 

'random': lambda: random.randint(1, 5), 

}) 

return env 


WARNING: Doii’t Alter jinja.EnvironmentAfter Instantiation 


While possible, modifying the j i nj a. Envi ronment object is dangerous. Per the Jinja2 API 
documentation, “Modifications on environments after the first template was loaded will lead 
to surprising effects and undefined behavior.” 

Reference: jinj a.pocoo.org/docs/dev/api/#jinj a2.Environment 


15.4 Resources 

>- Djangos documentation on usingJinja2: docs.djangoproject.com/en/1.11/topics/ 
templates/#dj ango.template.backends.jinj a2.3inj a2 
>- jinja.pocoo.org 

15.5 Summary 

In this chapter we covered the similarities and differences between DTL andjinja2. We also explored 
some of the ramifications and workarounds for using Jinja2 in projects. 

Starting in the next chapter we leave templates behind and explore the world of REST from both 
the server and client sides. 
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Building REST APIs With Django 
REST Framework 



Today’s internet is much more than HTML-powered websites. Developers need to support AJAX 
and native mobile apps. Having tools that support easy creation of JSON, YAML, XML, and other 
formats is important. By design, a Representational State Transfer (REST) Application Program- 
ming Interface (API) exposes application data to other concerns. 

The defacto package for building these REST APIs with Django is Django REST Framework 
(DRF). In fact, DRF has become so ubiqitious it’s not uncommon to hear questions like, “Whats the 
dilFerence between Django and Django REST Framework”? Since approximately 2013, we estimate 
90-95% of new Django projects with an API use DRF 

Why the popularity? WeU, we feel that the success of Django REST Framework is because: 

>- DRF leans heavily on object-oriented design and is designed to be easily extensible. 

>- DRF builds directly off of Django CBVs. If you understand CBVs, DRF’s design feels like an 
understandable extension of Django. 

>- It comes with a host of views for API generation, ranging from the 
djano . VI ews. generi c . View-like APIView to deep abstractions like generic API 
views and viewsets. 

>- The serializer system is extremely powerful, but can be trivially ignored or replaced. 

>- Authentication and Authorization are covered in a powerful, extendable way. 

>- If you really want to use FBVs for your API, DRF has you covered there too. 

Because of these reasons, DRF’s community is gigantic. This is important, because it means that many 
of the chaUenges in building REST APIs with it have been solved. Perhaps not in DRF directly, but 
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in third-party packages. Also, finding people who know it and can answer questions isn’t hard. 

We’ll go over the other side of REST APIs in Chapter 17: Consuming REST APIs. 

If you don’t know how to use DRF yet, we recommend its official tutorials: 

>- django-rest-framework.org/tutorial/quickstart/ 

>- django-rest-framework.org/tutorial/1-seri alization/ 


TIP: Django REST Framework NeedsYour Support! 


DRF is a coUaboratively funded project. If you use it commerciaUy we strongly encourage you 
to invest in its continued development by signing up for a paid plan. Thanks to this funding, 
weVe seen the project leap forward in terms of functionality. 

Financial contributions start at USS15 and go up. At the higher levels, the DRF project even 
provides priority support: fund .django- rest-framework.org/topi cs/funding/ 


16.1 Fundamentals of Basic REST API Design 

Let’s take a step back and look at HTTP and how it interacts with Django REST Framework. 


The Hypertext Transfer Protocol (HTTP) is a protocol for distributing content that provides a set of 
methods to declare actions. By convention, REST APIs rely on these methods, so use the appropriate 
HTTP method for each type of action: 


Purpose of Request 

HTTP Method 

Rough SQL equivalent 

Create a new resource 

POST 

INSERT 

Read an existing resource 

GET 

SELECT 

Update an existing resource 

PUT 

UPDATE 

Update part of an existing resource 

PATCH 

UPDATE 

Delete an existing resource 

DELETE 

DELETE 

Returns same HTTP headers as GET, but no 
body content 

HEAD 


Return the supported HTTP methods for the 
given URL 

OPTIONS 
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Echo back the request 

TRAGE 


Table 16.1: HTTP Methods 


A few notes on the above: 

>- If youre implementing a read-only API, you might only need to implement GET methods. 

>- If youre implementing a read-write API, you should use the GET, POST, PUT, and 
DELETE methods. 

>- Relying on just GET and POST for ali actions can be frustrating pattern for API users. 

>■ By definition, GET, PUT, and DELETE are idempotent. POST and PATCH are not. 

>- PATCH is often not implemented, but it’s a good idea to implement it if your API supports 
PUT requests. 

>- Django Rest Framework is designed around these methods, understand them and DRF itself 
becomes easier to understand. 

Here are some common HTTP status codes that should be considered when implementing a REST 
API. DRF’s generic views and viewsets return these values as appropriate for the method called. Note 
that this is a partial list; a much longer list of status codes can be found at en. wi ki pedi a . org/ 
wiki/List_of_HTTP_status_codes. 


HTTP Status Code 

Success/Failure 

Meaning 

200 OK 

Success 

GET - Return resource 

PUT - Provide status message or return 

resource 

201 Created 

Success 

POST - Provide status message or return 
newly created resource 

204 No Content 

Success 

PUT or DELETE - Response to successful 
update or delete request 

304 Not Modified 

Redirect 

ALL - Indicates no changes since the last 
request. Used for checking Last-Modified and 
ETag headers to improve performance. 
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HTTP Status Code 

Success/Failure 

Meaning 1 

400 Bad Request 

Failure 

ALL - Return error messages, including form 

validation errors. 

401 Unauthorized 

Failure 

ALL - Authentication required but user did not 
provide credentials or provided invalid ones. 

403 Forbidden 

Failure 

ALL - User attempted to access restricted 

content 

404 Not Found 

Failure 

ALL - Resource is not found 

405 Method Not Allowed 

Failure 

ALL - An unallowed HTTP method was 

attempted. 

410 Gone 

Failure 

ALL - A requested resource is no longer 
available and wont be available in the future. 

Used when an API is shut down in favor of a 
newer version of an API. Mobile applications 
can test for this condition, and if it occurs, teli 
the user to upgrade. 

429 Too Many Requests 

Failure 

ALL - The user has sent too many requests in a 
given amount of time. Intended for use with 
rate limiting. 


Table 16.2: HTTP Status Codes 


16.2 lUustrating Design Concepts With a Simple API 


In order to illustrate how DRF ties together HTTP methods, HTTP status codes, serialization, and 
views, let’s create a simple JSON API. WeT use the Jlavors app example from previous chapters as 
our base, providing the capability to create, read, update, and delete flavors via HTTP requests using 
AJAX, python-requests, or some other library. 


We’ll begin by checking that we have a tightly secured API. In our settings file we set our default 
permission classes to allow just admins: 
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TIP: IsAdminUser as the Constant Default Permission Class 


We like to lock down our projects, especially our REST APIs. There’s no better way to this 
then to have rest_f ramework. permi ssions. IsAdmi nUser as the default permission 
class. This we can override on a per-view basis. This makes our API views very secure by 
default, something worth more than having to add a few extra lines of code in our API 
views. 


With that out of the way, heres our Flavor model again, but enhanced with a UUID for API 
lookups: 
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WARNING: Doii’t Use Sequential Keys as Public Identifiers 


Sequential keys, such as what Django provides as a default as model primary keys, can be 
a security concern if used publicly. We cover this in-depth at Section 26.27: Never Display 
Sequential Primary Keys. 

In our example, we’re going to use the models UUID rather than the models primary key to 
look up our records. We always try to avoid using sequential numbers for lookups. 


Define the serializer class: 



Now let’s add in our API views: 
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permission_classes = (IsAuthenticated, ) 
seri alizer_class = FlavorSerializer 
lookup_field = 'uuid' # Don't use Flavor.id! 

class FlavorRetrieveUpdateDestroyAPIVi ew(Retri eveUpdateDestroyAPIView): 
queryset = Flavor.objects.all() 
permission_classes = (IsAuthenticated, ) 
seri alizer_class = FlavorSerializer 
lookup_field = 'uuid' # Don't use Flavor.id! 


Were done! Wow, that was fast! 


TIP: Classy Django REST Framework Is a Useful Reference 


For working with the Django Rest Framework, weVe found that http: //cdrf .co is a 
great cheat sheet. It is patterned after the famous ccbv .co.uk reference site, hut tailored 
for Django Rest Framework. 


Now we’U wire this into omx flavors/urh.py module: 
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vi ew=views.FlavorRetrieveUpdateDestroyAPIView.as_view(), 
name= 'flavor_rest_api' 

) 

] 


What we are doing is reusing the URLConf name, making it easier to manage when you have a need 
for a JavaScript-heavy front-end. Ali you need to do is access the Flavor resource via the {% uri %} 
template tag. 

In case it’s not ciear exactly what our URLConf is doing, let’s review it with a tahle: 


Uri 

View 

Uri Name (same) 

/flavors/api/ 

FlavorListCreateAPIV iew 

flavor_rest_api 

/flavors/api/: slug/ 

FlavorRetrieveUpdateDestroyAPIView 

flavor_rest_api 


Table 16.3: URLConf for the Flavor REST APIs 


WARNING: Our Simple API Does Not Use Permissions 


We overrode the default IsAdmi n permission with IsAuthenti cated. If you implement 
an API using our example, doht forget to assign user permissions appropriately! 

>- django-rest-framework.org/api-guide/authenti cation 
>- django- rest-framework.org/api-guide/permi ssions 


The end resuit is the traditional REST-style API definition: 
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TIP: Common Syntax for Describing REST APIs 


It’s not uncommon to see syntax like what is described in the Example 16.5: Wiring in API 
Views code example. In this particular case, / flavors/api / : uui d/ includes a : uuid value. 
This represents a variable, but in a manner suited for documentation across frameworks and 
languages, and you’ll see it used in many third-party REST API descriptions. 


WeVe shown you (if you didn’t know aiready) how it’s very easy to build REST APIs in Django. Now 
let’s go over some advice on maintaining and extending them. 

16.3 REST API Architecture 

Building simple, quick APIs is easy with Django REST Framework, but extending and maintaining 
it to match your projects needs takes a bit more thought. This is usually where people get hung up 
on API design. Here are some tips for improving your design: 


16.3.1 Use Consistent API Module Naming 

Just like anything else, how things are named needs to be consistent across a project. Our preferences 
for naming module related to API design is as foUows: 



Please observe the foUowing: 
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>■ We like to place ali our API components into a package within an app called api/ . That’s allows 
us to isolate our API components in a consistent location. If we were to put it in the root of 
our app, then we would end up with a huge list of API-specific modules in the general area of 
the app. 

>■ Viewsets belong in their own module. 

>- We always place routers in urls.py. 


16.3.2 Code for a Project Should Be Neatly Organized 

For projects with a lot of small, interconnecting apps, it can be hard to hunt down where a particular 
API view lives. In contrast to placing all API code within the each relevant app, sometimes it makes 
more sense to build an app specifically for the API. This is where aU the serializers, renderers, and 
views are placed. 

Of course, the name of the app should reflect its API version (see Section 16.3.7: Version Your API). 

For example, we might place aU our views, serializers, and other API components in an app titled 
apiv4. 

The downside is the possibility for the API app to become too large and disconnected from the apps 
that power it. Flence why we consider an alternative in the next subsection. 


16.3.3 Code for an App Should Remain in the App 

When it comes down to it, REST APIs are just views. For simpler, smaller projects, REST API 
views should go into views.py or viewsets.py modules and follow the same guidelines we endorse 
when it comes to any other view. The same goes for app- or model-specific serializers and renderers. 
Ifwe do have app-specific serializers or renderers, the same applies. 

For apps with so many REST API view classes that it makes it hard to navigate a single api/views.py 
or api/viewsets.py module, we can break them up. Specifically, we move our view (or viewset) classes 
into a api/views/ (or api/viewsets/) package containing Python modules typicaUy named after our 
models. So you might see: 
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flavors/ 

I— ap-i/ 

I I - __init__.py 

I I— ... other modules here 

I h- views 

I I I - __-init__.py 

I I I- flavor.py 

I I I- ingredient.py 


The downside with this approach is that if there are too many small, interconnecting apps, it can be 
hard to keep track of the myriad of places API components are placed. Hence why we considered 
another approach in the previous subsection. 


16.3.4 Try to Keep Business Logic Out of API Views 


Regardless of which architectural approach you take, it’s a good idea to try to keep as much logic as 
possible out of API views. If this sounds familiar, it should. We covered this in Section 8.5: Try to 
Keep Business Logic Out of Views. Remember, at the end of the day, API views are just another 
type of view. 
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Figure 16.1: An Ice Cream as a Service API. 


16.3.5 GroupingAPI URLs 

If you have REST API views in multiple Django apps, how do you build a project-wide API that 
looks like this? 


Example 16.9: Project-Wide API Design 


api/flavors/ # GET, POST 
api/flavors/;uuid/ # GET, PUT, DELETE 
api/users/ # GET, POST 
api/users/:uuid/ # GET, PUT, DELETE 


In the past, we placed all API view code into a dedicated Django app caUed api or apivl, with custom 
logic in some of the REST views, serializers, and more. In theory it’s a pretty good approach, but in 
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practice it means we have logic for a particular app in more than just one location. 

Our current approach is to lean on URL configuration. When building a project-wide API we write 
the REST views in the api/views.py or api/viewsets.py modules, wire them into a URLConf called 
something like core/api_urls.py or core/apivl_urls.py, and include that from the project roots urls.py 
module. This means that we might have something like the following code: 
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url( 

regex=r '''users/(?P<uuid> [-\w] +) / $ ' , 

vi ew=user_vi ews.UserReadUpdateDeleteView.as_view(), 

name= 'users' 

), 

] 


16.3.6 TestYourAPI 

We find that Django’s test suite makes it really easy to test API implementations. It’s certainly much 
easier than staring at curi results! Testing is covered at length in Chapter 22: Testing Stinks and Is 
a Waste of Money!. We even include in that chapter the tests we wrote for our simple JSON 7\PI 
(see Section 22.3.1: Each Test Method Tests One Thing). 

16.3.7 Version Your API 

It’s a good practice to abbreviate the uris of your API with the version number e.g. 
/api/vl/flavors or /api/vl/users and then as the API changes, /api/v2/flavors or 
/api /v2/users. When the version number changes, existing customers can continue to use the 
previous version without unknowingly breaking their calls to the 7\PI. 

Also, in order to avoid angering API consumers, it’s critical to maintain both the existing API and 
the predecessor API during and after upgrades. It’s not uncommon for the deprecated API to remain 
in use for several months. 

When we implement a new version of the API, we provide customers/users with a deprecation warn- 
ing along with ample time so they can perform necessary upgrades and not break their own applica- 
tions. From personal experience, the ability to send a deprecation warning to end users is an excellent 
reason to request email addresses from users of even free and open source API Services. 

16.3.8 Be Careful With Customized Authentication Schemes 

If youre building an API and need a customized authentication scheme, be extra careful. Security 
is hard, and there are always unpredictable edge cases, which is how people penetrate sites. We ve 
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only had to implement customized authentication scheme a few times, but we kept the following in 
mind: 

>- If we’re creating a new authentication scheme, we keep it simple and weU tested. 

>■ Outside of the code, we document why existing Standard authentication schemes are insuffi- 
cient. See the tipbox below. 

>- Also outside of the code, we document in depth how our authentication scheme is designed 
to Work. See the tipbox below. 

>- Unless we are writing a non-cookie based scheme, we don’t disable CSRF. 


TIP: Documentation Is Critical for Customized Authentication 


Writing out the why and how of a customized authentication scheme is a critical part of the 
process. Dont skip it! Heres why: 

>- Helps us validate our reasoning for coming up with something new. If we can’t describe 
the problem in writing, then we don’t fully understand it. 

>- Documentation forces us to architect the solution before we code it. 

>■ After the system is in place, later the documentation allows us (or others) to understand 
why we made particular design decisions. 


16.4 When DRF Gets in the Way 

Django Rest Framework is a powerful tool that comes with a lot of abstractions. Trying to work 
through these abstractions can prove to be extremely frustrating. Let’s take a look on overcoming 
them. 


16.4,1 Remote Procedure Calis vs REST APIs 

The resource model used by REST frameworks to expose data is very powerful, but it doesnt cover 
every case. Specifically, resources dont always match the reality of application design. For example, it 
is easy to represent syrup and a sundae as two resources, but what about the action of pouring syrup? 
Using this analogy, we change the state of the sundae and decrease the syrup inventory by one. While 
we could have the API user change things individually, that can generate issues with database integrity. 
Therefore in some cases it can begood idea topresent a method like sundae. pour_syrup (syrup) to 
the client as part of the RESTful API. 
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In computer Science terms, sundae. pour_syrup (sy rup) could be classified as a Remote Proce- 
dure CaU or RPC. 

References: 

>• en.wikipedia.org/wiki /Remote_Procedure_Call 

>- en.wikipedi a.org/wiki/Resource-oriented_architecture 

Fortunately, RPC calls are easy to implement with Django Rest Framework. AU we have to do is 
ignore the abstraction tools of DRF and rely instead on its base APIView: 



232 






16.4: When DRF Gets in the Way 


def get(self, request, *args, **kwargs) 

# Get list of syrups already poured onto the sundae 
sundae = get_object_or_404(Sundae, uuid=request . data[ 'uuid' ]) 
syrups = [SyrupSerializer(x) . data for x in sundae . syrup_set . ali()] 
return Response(syrups) 


And our API design would look like this now: 



16.4,2 Problems With Complex Data 

Okay, we’ll admit it, we make this mistake with DRF about once a month. Let’s sum up what happens 
in very simple terms with the following API design: 



O We have a model (Scoop) that we want represented within another (Cone) 

0 We can easily write a GET of the Cone that includes a list of its Scoops 
© On the other hand, writing a POST or PUT of Cones that also adds or updates its Scoops 
at the same time can be challenging, especially if it requires any kind of validation or post 
Processing 

O Frustration sets in and we leave to get some real-world ice cream 
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While there are nicely complex Solutions for nested data, weVe found a better solution. And that is 
to simplify things just a bit. Example: 

>• Keep the GET representation of the Cone that includes its Scoops 

>- Remove any capability of the POST or PUT for the Cone model to modify Scoops for that 
cone. 

>- Create GET/POST/PUT API views for Scoops that belong to a Cone. 

Our end API wiU now look like this: 



Yes, this approach does add extra views and additional API caUs. On the other hand, this kind of 
data modeling can resuit in simplification of your API. That simplification will resuit in easier testing, 
hence a more robust API. 

For what it’s worth, if you take a close look at the Stripe API reference (stri pe. com/docs/api) 
you’U see they foUow our pattern. You can view complex data, but you have to create it bit-by-bit. 


16.4,3 Simplify! Go Atomic! 

In the previous two subsections (RPC CaUs and Problems With Complex Data), we ve established a 
pattern of simplification. In essence, when we run into problems with DRF we ask these questions: 

>■ Can we simplify our views? Does switching to API Vi ew resolve the problem? 

>- Can we simplify our REST data model as described by views? Does adding more views (of a 
straightforward nature) resolve the problem? 

>- If a serializer is troublesome and is outrageously complex, why not break it up into two different 
serializers for the same model? 
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As you can see, to overcome problems with DRP, we break our API down into smaller, more atomic 
components. WeVe found that it’s better to have more views designed to be as atomic as possible 
than a few views with many options. As any experienced programmer knows, more options means 
more edge cases. 

Atomic-style components help in these regards: 

>- Documentation is easier/faster because each component does less 
>- Easier testing since there are less code branches 

>- Bottlenecks are easier to resolve because chokepoints are more isolated 
>- Security is better since we can easily change access per view rather than within the code of a 
view 


16.5 Shutting Down an External API 

When it’s time to shut down an older version of an external API in favor of a new one, here are useful 
steps to follow: 


16.5.1 Step #1: Notify Users of Pending Shut Down 

Provide as much advanced notice as possible. Preferably six months, but as short as one month. 
Inform API users via email, blogs, and social media. We like to report the shutdown notification to 
the point that we worry people are getting tired of the message. 


16.5.2 Step #2: Replace API With 410 Error View 

When the API is finally shut down, we provide a simple 410 Error View. We include a very simple 
message that includes the following Information: 

>- A link to the new APPs endpoint. 

>- A link to the new APPs documentation. 

>- A link to an article describing the details of the shut down. 

Below is a sample shutdown view that works against any HTTP method: 
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16.6 Rate-LimitingYour API 

Rate limiting is when an API restricts how many requests can be made by a user of the API within 
a period of time. This is done for a number of reasons, which we’ll explain below. 


16.6.1 Unfettered API Access Is Dangerous 

In the ancient times (2010) we launched the djangopackages.org website. The project, started 
during the Django Dash contest, was an instant hit for the Django community. We sprinted on it 
constantly, and its feature set grew rapidly. Unfortunately, we quickly hit the rate limit of GitHub’s 
first API. This meant that after a certain amount of API requests per hour, we werent aUowed to 
make any more until a new hour passed. 

Fortunately at DjangoCon 2010 we had the opportunity to ask one of the founders of GitHub if we 
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could have unlimited access to their API. He graciously said ‘yes’ and within a day we could get data 
from GitHub as much as we wanted. 

We were delighted. Our users were delighted. Usage of the site increased, people were thirsty for data 
as to what were the most active projects. So desirous of data were we that every hour we requested 
the latest data from GitHuh. And that caused a problem for GitHub. 

You see, this was 2010 and GitHub was not the giant, powerful company it is today. At 17 minutes 
past each hour, Django Packages would send thousands of requests to the GitHub API in a very 
short period. With unfettered access we were causing them problems. 

EventuaUy, GitHub contacted us and requested that we scale back how much we were using their 
API. We would stili have unlimited access, just needed to give them breathing room. We complied, 
checking data once per day instead of by the hour, and at a more reasonable rate. We continue to do 
so to this day. 

While modern GitHub can certainly handle much, much larger volumes of API access then it could 
in late 2010, we like to think we learned a shared lesson about unfettered access to an API: Grant 
such access cautiously. 

16.6.2 REST Frameworks Must Come With Rate Limiting 

ControUing the volume of REST API access can mean the dilference between joyful triumph or 
utter disaster in a project. 


TIP: HTTP Server Rate Limiting 


It’s possible to use nginx or apache for rate limiting. The upside is faster performance. The 
downside is that it removes this functionality from the Python code. 


16.6.3 Rate Limit Can Be a Business Plan 

Imagine we launch an APTbased startup that lets users add images of toppings to images of ice 
cream. We know that everyone wiU want to use this API, and come up with several tiers of access 
that we tie into pricing: 
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Developer tier is free, but only allows 10 API requests per hour. 

One Scoop is $24/month, allows 25 requests per minute. 

Two Scoops is $79/month, allows 50 requests per minute. 

Corporate is $5000/month, allows for 200 requests per minute. 

Now ali we have to do is get people to use our API. 

16.7 AdvertisingYour REST API 

Let’s assume weVe built our REST API and want outside coders and companies to use it. How do 
we go about doing that? 


16.7.1 Documentation 

The most important thing to do is to provide comprehensive documentation. The easier to read 
and understand the better. Providing easy-to-use code examples is a must. You can write it 
from scratch or use auto-documentation tools provided by django-rest-framework itself and vari- 
ous third-party packages (django-rest-framework. org/topi cs/documenti ng-your-api / 
#thi rd- pa rty- packages. You can even embrace a commercial documentation generation Services 
like readthedocs. com and swagger. i o. 

Some of the material in Chapter 23: Documentation: Be Obsessed might prove useful for forward- 
facing REST API documentation. 


16.7.2 Provide Client SDKs 

Something that may help spread use of your API is to provide a Software development kits (SDK) 
for various programming languages. The more programming languages covered the better. For us, 
weVe found the must-have languages include Python, JavaScript, Ruby, PHP, Go, andjava. 

In our experience, it’s a good idea to write at least one of these libraries ourselves and create a demo 
project. The reason is that it not only advertises our API, it forces us to experience our API from the 
same vantage point as our consumers. 
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16.8: Additional Reading 


Fortunately for us, thanks to the underlying Core API document object model (coreapi .org), 
DRF provides several JSON Hyperschema-compatible libraries. When used client-side, Core API 
allows for dynamically driven client libraries that can interact with any API that exposes a supported 
schema or hypermedia format. 

These tools should work immediately with most DRF-powered APIs: 

>- Command line client 

django-rest-framework.org/topics/api-clients/#command-line-Client 
>■ Python client library 

django-rest-framework.org/topics/api-clients/#python-Client-library 
>- JavaScript client library 

django-rest-framework.org/topi cs/api-clients/ 

#javascript-client-library 

For building Python-powered client SDKs, reading Section 21.9: Releasing Your Own Django Pack- 
ages might prove useful. 


16.8 Additional Reading 

We highly recommend reading the following: 

>- en.wikipedia.org/wiki/REST 
>• coreapi . org 

>- en.wikipedi a.org/wiki/List_of_HTTP_status_codes 
>- j acobi an.org/writi ng/rest-worst-practices/ 


PACKAGE TIP: Other Packages for Crafting APIs 


For the reasons explained at the beginning of this chapter, we recommend Django Rest Frame- 
work. Flowever, should you for some unfathomable reason choose not to use DRF, consider 
the following packages: 

>- django-tastypie is a mature API framework that implements its own class-based view 
System. Predating Django REST Framework, it’s a feature-rich, mature, powerful, sta- 
ble tool for creating APIs from Django models. 
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Figure 16.2: A tasty pie is one filled with ice cream. 

>- For super-quick, super-simple one-off REST API views django-braces (CBVs) and 
django-jsonview (FBVs) can prove really useful. The downside is that when you get 
into the flill range of HTTP methods and more complex designs, they rapidly become 
a hindrance due to their lack of focus on building APIs. 


16.9 Summary 

In this chapter we covered: 

>- Why you should use Django Rest Framework 

>- Basic REST API concepts and how they relate to Django Rest Framework 
>- Security considerations 
>- Grouping strategies 
>• Simplification strategies 
>- Fundamentals of basic REST API design 

Corning up next, weT go over the other side of REST APIs in Chapter 17: Consuming REST APIs. 
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Consuming REST APIs 



Now that weVe covered both creating REST APIs and template best practices, let’s combine them. 
In other words, these are best practices for using Django-powered tools to display content to the 
end User in the browser using content managed by REST APIs and presented by modern JavaScript 
frameworlcs. 


WARNING: This Chapter Will Be Brief 


Our chaUenge in writing this chapter is twofold: 

>- Django is a backend framework. 

>■ The modern JavaScript/HTMLS landscape is evolving too fast to document. The tech- 
nical term for trying to keep up with it is called JavaScript Fatigue. 

Therefore, we’re going to cover best practices at a very high level. 


With the advent of faster JavaScript engines and a maturation of the associated community, there 
has been a rise in new JavaScript frameworks that are designed for integration with REST APIs. The 
popular ones at the start of 2017 seem to be: 

React.js facebook. github. io/react/ 

A JavaScript framework and ecosystem created and, for the moment, maintained by Facebook. 
Designed for creation of HTML, iOS, and Android applications. 

Vue.js vuejs.org 

Rapidly rising in popularity, Vue.js promises to be simpler to execute than React. The youngster 
of this list, its ecosystem isn’t as large as the others. 
jQuery jquery.com 

While it hasnt been on the JavaScript hipster list for years, 75 percent of the web eant be 
wrong. jQuery’s ecosystem is gigantic. 
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These libraries can really improve what we like to call the ‘immediate user experience’. However, with 
every good thing there are always things to consider and things to do. 


17.1 Learn How to Debug the Client 


Debugging client-side JavaScript is a lot more than simply writing console. log() and con¬ 
scie . di r () statements. There are a number of tools for debugging and finding errors, and some 
of them are specifically written for particular JavaScript frameworks. Once a tool is chosen, it’s an 
excellent idea to take a day to learn how to write client-side tests. 


Reference material: 


>- developers.google.com/web/tools/chrome-devtools 
>• developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging 
JavaScript 




Figure 17.1: Server-side vs. client-side ice cream. 
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17.2: Consider Using JavaScript-Powered Static Asset Preprocessors 


17.2 Consider Using JavaScript-Powered Static Asset Preproces¬ 
sors 

In the past, we used Python everywhere, including JavaScript and CSS minification. However, these 
days it’s ciear that the JavaScript community is maintaining their versions of these tools better than 
the Python community. That’s perfectly okay, because since theyve done the work on this part of the 
toolchain, we can focus on other things. 

As we write this, the most commonly used tool for this kind of work is...debatable. We’re not even 
going to list the options because by the time you read this, we are 100% certain some of those options 
wiU be out of date. We submit that you pick the one that appears to have the most traction at the time 
you read this paragraph. Fortunately, as node.js works everywhere Python does (including Windows), 
you won’t have a problem running whatever you choose. 

17.3 Real-Time Woes a.k.a. Latency 

Let’s say weVe put together a well-designed, weU-indexed, well-cached real-time project with the 
widest bandwidth piping content to the world. We can handle any load, and our test users applaud 
the speed and functionality of the project. Things look great, and we look forward to bonuses and 
raises. 

Then the complaints from the other side of the planet start coming in about the slow speed of the 
application. Our elfort isht ‘real-time’ to any of a potentially large block of users and our client/boss 
is really unhappy. 

Congratulations, we’vejust hit the speed oflight! 

This isht a joke, it’s a very real problem. Here, Django isht the problem. Instead, it’s physics. The time 
it takes for HTTP requests to transmit back and forth across half the circumference of the planet 
is noticeable to human beings. Add in server-side and client-side processing, and we risk alienating 
potential or existing users. 

Also, keep in mind that even the fastest local connections have hiccups and slow-downs. So it’s not 
uncommon for ‘real-time’ applications to have ways to handle this sort of behavior. 
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17.3.1 Solution: Mask the Latency With Animations 

One of the more common fixes is to have JavaScript-powered animation distract the user from latency 
issues. We encounter this every time we use a single page app with an attractive interface, including 
all modern web-based email clients. 


17.3.2 Solution: Fake Successful Transactions 

Another solution involves processing the request on the client-side as if the request successfully made 
it to the server. We’ll need to include client-side logic to handle failures, but JavaScript frameworks 
handling HTTP requests are asynchronous, making this feasible, albeit possibly complicated. 

If youVe ever suddenly discovered that your cloud-based spreadsheet hadht save the data entered for 
the past 30 seconds, you ve uncovered this kind of JavaScript powered trickery in action. As this can 
be very frustrating, some Online tools upon detecting a connection failure, disallow further data entry. 


17.3.3 Solution: Geographically Based Servers 

Geographically-based servers across aU seven continents is an option. However, for Django this is 
not trivial to implement, not at the programming or database level. It requires a significant volume 
of skills and expertise that’s outside the scope of this book. 

If you have the time and budget, this can be an exciting avenue to explore and we encourage it. 
However, unless you ve done this before there is a good chance you are going to underestimate the 
effbrt involved. 


17.3.4 Solution: Restrict Users Geographically 

Sometimes we just don’t have a choice. Perhaps our application is too reliant on ‘real-time’ perfor- 
mance and geolocating servers might be outside the budget. We might make some people unhappy, 
but that can be mitigated to some degree by saying things like, ‘Support in your country is coming 
soon!’ 
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17.4 Avoid the Anti-Patterns 

Here are a number of anti-patterns that weVe discovered when it comes to projects consuming REST 
APIs for content. 


17.4.1 Building Single Page Apps When Multi-Page Apps Suffice 

Single-page apps are challenging and fun to build, but does a traditional CMS-site need to be one? 
Certainly the content pages can include API-powered editing Controls, but when building this kind 
of site, there is something to be said for traditional HTML pages. 

For example, our health provider has a SPA-style site. It’s absolutely lovely. The way everything 
moves together is a marvel. And it’s completely useless when you have to do any kind of comparative 
research. 

The worst example of the site is that when the search system returns a list of doctors you can’t easily 
compare them. When you click on one for more information, their data is in a sliding modal. You 
eant right-click and open several on independant tabs as doing so just takes you to the root search 
page. You can print or email yourself the information on individual doctors, but PDFs and email are 
awful comparison tools compared to hopping between tabs. 

What the site should provide is individual domain references (i.e. URLs) for each doctor. Either 
parsed by the server on the back end or even by JavaScript URL management on the front end. This 
isn’t hard to do, yet it remains a painfuUy common issue. 


17.4,2 Upgrading Legacy Sites 


Unless the entire site is being scrapped for a new version, don’t upgrade the whole front-end at once. 

When working with legacy projects, it’s often easier to add new features as single-page apps. This 
allows for the maintainers of the project to deliver improved experiences with new features, while 
preserving the stability of the existing code base. A good example of this might be adding a calendar 
application to an existing project. 
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17.4.3 Not WritingTests 

When you first begin working in a new language or framework, including client-side JavaScript, it’s 
tempting to skip the tests. In a word, don’t. Working in the client is getting more complicated and 
sophisticated every year. Between evolving client-side standards, things are simply not as readable 
there as on the server side. 

We cover Django/Python testing in Chapter 22: Testing Stinks and Is a Waste of Money!. 
A good reference for JavaScript testing is stackoverflow.com/questions/300855/ 
j avascript-unit-test-tools-for-tdd 

17.4.4 Not Understanding JavaScript Memory Management 

Single-page apps are great, but the complex implementations where users keep them open constantly 
wiU hold objects in the browser for a very long time. Eventually, if not managed, this can cause browser 
slowdowns and crashes. Each JavaScript framework comes with tools or advice on how to handle this 
potential problem, and it’s a good idea to know the recommended approach. 


17.4.5 Storing Data in the DOM When It’s Not jQuery 

After years of using jQuery, some of us have grown used to using DOM elements to store data 
(especially Daniel). However, when using other JavaScript frameworks this isnt ideal. They have 
their own mechanisms for handling client data, and by not foUowing them we risk losing out on 
some of the features promised by these frameworks. 

We recommend looking up the data management methods for your chosen JavaScript framework 
and embracing them as deeply as possible. 

17.5 AJAX and the CSRFToken 

Djangos CSRF protection appears to be an inconvenience when writing AJAX. Ifyou use AJAX with 
Django, you may discover that triggering the CSRF token validation blocks your ability to POST, 
PATCH, or DELETE data to your API. However, it’s part of what makes Django secure, dont 
disable it! 
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Our answer to overcoming the CSRP hurdle is as foUows: 

O For the hackend, always use Django REST Framework whenever we have an API that handles 
POST, PATCH, or DELETE requests. 

0 On the front-end, we recommend using Django REST Framework’s huilt-in 
JavaScript client lihrary (django-rest-framework.org/topics/api-clients/ 
#javascript-client-library) to interface with the hackend. It’s framework agnostic 
and trivially handles the CSRF token for you. 

© Sometimes we want a tighter integration than what the DRF client framework provides. In 
these cases it’s paramount for us to continue relying on the CSRF framework. 

References: 

>• django-rest-framework.org/topi cs/api-clients/ 

#javascript-client-library 
>• docs.dj angoproj ect.com/en/1.11/ref/csrf/ 

>- github.com/GetBlimp/django-rest-framework-jwt This is a proven authentication 
lihrary for DRF that is safe to use without CSRF 
>- Section 17.5.1: Set settings.CSRF_COOKIE_HTTPONLY Appropriately 


TIP: What ifYou Aren’t Using Django REST Framework 


Then youre going to have to figure out how to handle CSRF on your own. A good reference is 
the official Django documentation on handling CSRF with AJAX: docs. dj angoproj ect. 
com/en/1.ll/ref/csrf/#ajax 


WARNING: Don’t Use AJAX as an Excuse to Turn Off CSRF 


Django core developer Aymeric Augustin says, “...CSRFprotection is almost always disabled 
because the developers couldht quite figure out how to make it work. It’s fine to disable CSRF 
if the API oniy accepts JWT authentication; it’s wrong it if accepts cookie authentication.” 
Unless you are using django-rest-framework-jwt (dj ango-rest-framework-JWT), dont 
build a site with disabled CSRF If you can’t figure out how to make it work, ask for help. No 
one’s going to make fun of someone trying to make their site more secure. 


247 












Chapter 17: Consuming RESTAPIs 


17.5.1 Set settings.CSRF_COOKIE_HTTPONLY Appropriately 


By setting the CSRF_COOKIE_HTTPONLY token to True we make it harder for malicious 
JavaScript to bypass CSRF protection. On the downside, you can’t use JavaScript to pull the CSRF 
token from the cookie. Therefore, per Djangos instructions, you have to pull the hidden CSRF token 
form input from the page. Here’s a jQuery-based example: 




17.6 ImprovingJavaScript Skills 


One of the best things we can do when implementing the consumption of REST APIs on the client 
side is to ensure our JavaScript skills are up to par. While Python developers sometimes like to 
grumble about JavaScript, it is a very capable language in its own right. Any responsible web developer 
wiU take the time to ramp up their skills so they can reap the benefits of modern JavaScript. 


248 











17.7: Follow JavaScript Coding Standards 


17.6.1 Assessing Skill Levels 

Noted JavaScript developer Rebecca Murphey created a JavaScript assessment tool. We found it a 
wonderfulway to determine how much JavaScript we actually knew, and what we needed to improve. 

See gi thub.com/rmurphey/ j s-assessment. 


17.6.2 Leam More JavaScript! 

There are plenty of resources available for improving your basic JavaScript skills. We list our favorites 
at the end of Appendix C: Additional Resources. 

17.7 Follow JavaScript Coding Standards 

In the case of JavaScript, we advocate the following guides for both front- and back-end work: 

>- Felix’s Node.js Style Guide 
nodeguide.com/style.html 
>- idiomatic.js 

github.com/rwaldron/idiornatic.j s 


17.8 Summary 

Material covered in this chapter included: 

>- Debugging the client. 

>- JavaScript static asset preprocessors. 
>- Real-time woes. 

>- Client-side anti-patterns. 

>• AJ/IX and CSRF tokens. 

>• Improving JavaScript skiUs. 

>• Useful resources. 
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18 I TradeofFs of Replacing 
Core Components 


Theres a lot of hype around swapping out core parts of Djangos stack for other pieces. Should you 

do it? 

Short Answer: Don’t do it. Even the CEO of Instagram (Kevin Systrom) said Forbes.com that it’s 
completely unnecessary (bi t . ly/2pZx0B0). 

LongAnswer: It’s certainly possible, since Django modules are simply just Python modules. Is it 
worth it? WeU, it’s worth it only if: 

>- You are okay with sacrificing some or aU of your abibty to use third-party Django pack- 
ages. 

>- You have no problem giving up the powerful Django admin. 

>- You have already made a determined effort to build your project with core Django com¬ 
ponents, but you are running into walls that are major blockers. 

>- You have already analyzed your own code to find and fix the root causes of your problems. 
For example, youVe done aU the work you can to reduce the numbers of queries made in 
your templates. 

>- Youve explored all other options including caching, denormalization, etc. 

>- Your project is a real, live production site with tons ofusers. In other words, youre certain 
that youre not just optimizing prematurely. 

>■ Youve looked at and rejected adopting a Service Oriented Approach (SOA) for those 
cases Django has problems dealing with. 

>- Youre willing to accept the fact that upgrading Django will be extremely painful or im- 
possible going forward. 

That doesnt sound so great anymore, does it? 
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18.1 The Temptation to Build FrankenDjango 

Every year, a new fad leads waves of developers to replace some particular core Django component. 
Here’s a summary of some of the fads weVe seen come and go. 


Fad Reasons 


For performance reasons, 
replacing the 

database/ORM with a 

No SQL database and 
corresponding ORM 
replacement. 

Not okay: “I have an idea for a social network for ice cream haters. 

I just started building it last month. I need it to be web-scale!!!l!” 

Okay: “Our site has 50M users and Lm hitting the limits of what I 
can do with indexes, query optimization, caching, etc. We’re also 
pushing the limits of our Postgres cluster. IVe done a lot of 
research on this and am going to try storing a simple denormalized 
view of data in Cassandra to see if it helps. Lm aware of the CAP 
theorem (en. wi ki pedi a. org/wi ki /CAP_theorem), and for 
this view, eventual consistency is fine.” 

For data processing 
reasons, replacing the 
database/ORM with a 

No SQL database and 
corresponding ORM 
replacement. 

Not okay: “SQL Sucks! We’re going with a document-oriented 
database like MongoDB!” 

Okay: “While PostgreSQLs HSTORE datatype replicates nearly 
every aspect of MongoDB’s data storage system, we want to use 
MongoDB’s built-in MapReduce functionality." 

Replacing Djangos 
template engine with 
Jinja2, Mako, or 
something else. 

Not okay: “I read on Hacker News that Jinja2 is faster. I don’t 
know anything about caching or optimization, but I need Jinja2!” 

Not okay: “I hate having logic in Python modules. I want logic in 
my templates!” 

Okay: “I have a small number of views which generate 1MB+ 
HTML pages designed for Google to index. Lll use Djangos 
native support for multiple template languages to render the 

1MB+ sized pages with Jinja2, and serve the rest with Django 
Template Language.” 


Table 18.1: Fad-based Reasons to Replace Components of Django 
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18.2: Non-RelationalDatabases vs. Relational 

Databases 



Figure 18.1: Replacing more core components of cake with ice cream seems like a good idea. Which 
cake would win? The one on the right, of course. 


18.2 Non-Relational Databases vs. Relational 
Databases 

Even Django projects that use relational databases for persistent data storage rely on non-relational 

databases. If a project relies on tools like Memcached for caching and Redis for queuing, then it’s 

using non-relational databases. 

The problem occurs when NoSQL Solutions are used to completely replace Django’s relational 

database functionality without considering in-depth the long-term implications. 

18.2.1 Not Ali Non-Relational Databases Are ACID Compilant 

ACID is an acronym for: 

Atomicity means that all parts of a transaction work or it all fails. Without this, you risk data cor- 
ruption. 

Consistency means that any transaction will keep data in a valid state. Strings remain strings and 
integers remain integers. Without this, you risk data corruption. 

Isolation means that concurrent execution of data within a transaction will not collide or leak into 
another transaction. Without this, you risk data corruption. 
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Durability means that once a transaction is committed, it will remain so even if the database server 
is shut down. Without this, you risk data corruption. 

Did you notice how each of those descriptions ended with 'Without this, you risk data corruption. ’? This 
is because in the case of most NoSQL engines, there is little-to-no mechanism iorACID compliance. 
It’s much easier to corrupt the data, which is mostly a non-issue for things like caching but another 
thing altogether for projects handling processing of persistent medical or e-commerce data. 


18.2.2 Don’t Use Non-Relational Databases for Relational Tasks 

Imagine if we were to use a non-relational database to track the sale of properties, property owners, 
and how property laws worked for them in 50 US States. There are a lot of unpredictable details, so 
wouldnt a schemaless datastore be perfect for this task? 

Perhaps... 

We would need to track the relationship between properties, property owners, and laws of 50 States. 
Our Python code would have to maintain the referential integrity between aU the components. We 
would also need to ensure that the right data goes into the right place. 

For a task like this, stick with a relational database. 


18.2.3 Ignore the Hype and Do Your Own Research 

It’s often said that non-relational databases are faster and scale better than relational databases. 
Whether or not this is true, dont blindly swallow the marketing hype of the companies behind 
any particular alternative database solution. 

Instead, do as we do: search for benchmarks, read case studies describing when things went right or 
wrong, and form opinions as independently as possible. 

Also, experiment with unfamiliar NoSQL databases on small hobby side projects before you make 
major changes to your main project infrastructure. Your main codebase is not a playground. 

Lessons learned by companies and individuals: 
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> Pinterest: medi um. com/@Pi nterest_Engi neeri ng/stop-usi ng-shi ny-3el613c2cel4 
>- DanMcKinleywhile at Etsy: mcfunley. com/why-mongodb-never-worked-out-at-etsy 

18.2.4 How We Use Non-Relational Databases With Django 

This is how we prefer to do things: 

>- If we use a non-relational data store, limit usage to short-term things like caches, queues, and 
sometimes denormalized data. But avoid it if possihle, to reduce the numher of moving parts. 

>- Use relational data Stores for long-term, relational data and sometimes denormalized data (Post- 
greSQUs array and HStore fields work great for this task). 

For us, this is the sweet spot that makes our Django projects shine. 

18.3 What About Replacing the Django Template 
Language? 

We advocate the practice of sticking entirely to the Django Template Language (DTL) with the 
exception of rendered content of huge size. However, as this use case is now covered hy Django’s 
native support of alternate template systems, weVe moved discussion of this topic to Chapter 15: 
Django Templates and Jinja2. 

18.4 Summary 

Always use the right tool for the right joh. We prefer to go with stock Django components, just like 
we prefer using a scoop when serving ice cream. However, there are times when other tools make 
sense. 

Just don’t follow the fad of mixing vegetahles into your ice cream. You simply can’t replace the classic 
strawberry, chocolate, and vanilla with supposedly “high-performance” flavors such as broccoli, corn, 
and spinach. Thats taking it too far. 


255 




Chapter 18: Tradeojfs ofRepIacing Core Components 


256 



19 Working W ith the Dj ango Admin 


When people ask, “What are the benefits ofDjango over other web frameworksf” the admin is what 
usually comes to mind. 


Imagine if every gallon of ice cream came with an admin interface. Youd be able to not just see the 
list of ingredients, but also add/edit/delete ingredients. If someone was messing around with your 
ice cream in a way that you didnt like, you could limit or revoke their access. 



Pretty surreal, isnt it? Well, that’s what web developers coming from another background feel like 
when they first use the Django admin interface. It gives you so much power over your web application 
automaticaUy, with little work required. 



Chapter 19: Working With the Django Admin 


19.1 It’s Not for End Users 

The Django admin interface is designed for site administrators, not end users. It’s a place for your site 
administrators to add/edit/delete data and perform site management tasks. Although it’s possible to 
stretch it into something that your end users could use, you really shouldnt. It’s just not designed for 
use by every site visitor. 


19.2 Admin Customization vs. New Views 

It’s usually not worth it to heavily customize the Django admin. Sometimes, creating a simple view 
or form from scratch results in the same desired functionality with a lot less work. WeVe always had 
better results with creating custom management dashboards for client projects than we have with 
modifying the admin to fit the need of the client. 


19.3 Viewing String Representations of Objects 

The default admin page for a Django app shows a list of generic looking objects like this: 


Django administration 

WELCOME, AUDREYR. VIEW SITE / CHANGE PASSWORD / LOG OUT 

Home Icecreambars ■ ico cream b«rs 



O The ice cream bar "IceCreamBar object" was changed successfully. 


Select ice cream bar to change 

Aetion: - t Go 0 of 3 selected 

L . ICE CREAM BAR 
Q IceCreamBar object 
□ IceCreamBar object 
Q IceCreamBar object 
3 ice cream bars 


Figure 19.2: Admin list page for an ice cream bar app. 
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19.3: Viewing String Representations of Objects 


That’s because the default string representation of an IceCreamBar object is “IceCreamBar object". 
Wouldnt it be helpful to display something better? 



19.3.1 Using_str_() 


Implementing_str_() is simple: 



The resuit is as foUows: 
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DjSHQO administration WELCOME,AUDREYn VIEWSITE/CHANGEPASSWORD/LOG out 

Home Icecreambare 


Select ice cream bar to change 

Aetion: - t [ Go ■ OofSselected 

ICE CREAM BAR 
B Vanilla Crisp 
6 MintCookieCrunch 
Q Strawberry Pie 
3 ice cream bars 


Figure 19.4: Improved admin list page with better string representation of our objects. 


c 


It’s more than that. When youre in the shell, you see the better string representation: 



The_str_() method is called whenever you call str () on an object. This occurs in the Django 

shell, templates, andbyextension the Django admin. Therefore, try to make the results of_str_() 

nice, readable representation of Django model instances. 


19.3.2 Using list_cHsplay 


If you want to change the admin list display in a way that isn’t quite a string representation of the 
object, then use li st_di splay. 
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The resuit with the specified fields: 


Django administration 

WELCOME, AUDREYR. VIEW SITE / CHANGE PASSWORD / LOG OUT 

Home Icecreambars i: 



Select ice cream bar to change 


Aetion: - % [ Go j 0 of 3 selected 

6 NAME SHELL FILLING 

B VaniliaCrisp milkchocolate with crisped rice vanilla ice cream 

□ Mint Cookie Crunch dark chocolate with cookie bits mint chip ice cream 

B Strawberry Pie white chocolate with graham crackercrust strawberry 

3 ice cream bars 


Figure 19.5: Further improvements to the admin list page. 


19.4 Adding Callables to ModelAdmin Classes 

You can use callables such as methods and functions to add functionality to the Django 
dj ango. contri b. admi n. ModelAdmi n class. This aUows you to really modify the list and display 
screens to suit your ice cream project needs. 

For example, it’s not uncommon to want to see the exact URL of a model instance in the Django 
admin. If you define a get_absolute_u rl () method for your model, what Django provides in the 
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admin is a link to a redirect view whose URL is very different from the actual object URL. Also, there 
are cases where the get_absolute_url () method is meaningless (REST APIs come to mind). 


In the example below, we demonstrate how to use a simple callable to provide a link to our target 

URL: 



Since a picture is worth a thousand words, here is what our callable does for us: 
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Django administration 

WELCOME. AUDREYR. VIEW SITE / CHANGE PASSWORD / L06 OUT 

Home > Icecreambars ’ Ice cream bars i" 


Change ice cream bar 

Name: What? 

Shell: Cinnamon 

Filling: Minced jalapeho and pesto 



B Has stick 

Ice Cream Bar URL: /icecreambars/4/ 


Delete 


Save and add another I Save and continue editing I SAVE 


Figure 19.6: Displaying URL in the Django Admin. 


WARNING: Use the allow_tags Attribute With Caution 


The allow_tags attribute, which is set to False by default, can be a security issue. When 
allow_tags is set to True, HTML tags are allowed to be displayed in the admin. 

Our hard rule is allow_tags can only be used on system generated data like primary keys, 
dates, and calculated values. Data such as character and text fields are completely out, as is 
any other user entered data. 


19.5 Be Aware of the Complications ofMultiuser Environments 

Nothing in the Django admin locks records to a particular staiF- or admin-level user. While this is 
fine for a project with a single person with admin-level access, on a multi-user project it can be a very 
serious problem. Here is what happens: 

O Via the Django admin, Daniel edits the record for “Peppermint Sundae” ice cream bar. He 
starts to make changes. He gets a phone caU from the marketing officer of Icecreamlandia and 
leaves his screen open. 
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& In the meantime, Audrey decides to modify “Peppermint Sundae” ice cream bar. She spends 
ten minutes making her changes, then saves the her data. 

© Daniel gets off the phone and finally saves his changes. He overwrites Audrey’s changes. 

If you have multiple users with access to the Django admin, you need to be aware of this possibility. 

19.6 Django’s Admin Documentation Generator 

One of the more interesting developer tools that Django provides is the 
dj ango. contri b. admi ndocs package. Created in an era before the advent of the docu¬ 
mentation tools that we cover in Chapter 23: Documentation: Be Obsessed, it remains a useful 
tool. 

It’s useful because it introspects the Django framework to display docstrings for project components 
like models, views, custom template tags, and custom filters. Even if a project’s components don’t 
contain any docstrings, simply seeing a list of harder-to-introspect items like oddly named custom 
template tags and custom filters can be really useful in exploring the architecture of a complicated, 
existing application. 

Using d j ango. cont ri b. admi ndocs is easy, but we like to reorder the steps described in the formal 
documentation: 

O pip install docuti Is into your project’s virtualenv. 

0 Add django.contrib.admindocs to your INSTALLED_APPS. 

© Add ( r '''admin/doc/ ' , i nclude (' dj ango. contri b . admi ndocs. uris ')) to your 
root URLConf. Make sure it’s included before the r ' ''admin/ ' entry, so that requests to 
/admin/doc/ don’t get handled by the latter entry. 

O Optionah Using the admindocs bookmarklets requires the XVi ewMi ddlewa re to be instaUed. 

Once you have this in place, go to /admin/doc/ and explore. You may notice a lot of your 
project’s code lacks any sort of documentation. This is addressed in the formal documentation on 
django.contrib.admindocs: docs.djangoproject.com/en/1.11/ref/contrib/admin/ 
admi ndocs/ and our own chapter on Chapter 23: Documentation: Be Obsessed. 

19.7 Using Custom Skins With the Django Admin 

Over the years there have been a number of efforts to reskin or theme the Django Admin. These 
range from the venerable, stable, and very popular django-grappeUi to more recent up-and-comers. 
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They allow easy-to-hard customization. 


PACKAGE TIP: Custom django.contrib.admin Skins 


Here are some of the more popular custom skins that are generally Python 2 and 3 compatible: 
>- django-grappelli is the grand-daddy of aU custom Django skins. Stable, robust, and 
with a unique but friendly style. 

>- django-suit is a relatively recent project and like many modern custom Django skins, 
is built using the familiar Twitter Bootstrap front-end framework. 

>- django-admin-bootstrapped is another port of the Django admin to Twitter Boot¬ 
strap. 

A more complete list can be found at 
dj angopackages.org/grids/g/admin-styling/. 


Django has a gigantic community, so why arent there more skins? 

It turns out that besides the most basic CSS-based modifications, creating custom Django themes 
is very chaUenging. For anyone who has delved into the source code for these projects, it’s 
ciear that custom admin skins require arcane code to account for some of the idiosyncrasies of 
django.contrib.admin. 

Patrick Kranzhnueller, maintainer of django-grappelli, goes into great detail in his article on the 
subject, A Frontend Framework for the Django Admin Interface’, which you can read at the link 
below: 

>• sehmaschine.net/blog/dj ango-admin-frontend-framework. 

Here are some tips when working with custom d j ango. contri b. admi n skins: 

19.7.1 Evaluation Point: Documentation is Everything 

As mentioned earlier, writing a custom skin for django. contri b. admi n is hard. While the suc- 
cessful skins are relatively easy to add to a project, it’s the edge cases (invariably involved in extending 
the ModelAdmi n object) that can hurt. 

Therefore, when evaluating one of these projects for use on a project, check to see how far the docu¬ 
mentation goes beyond instaUation instructions. 
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19.7.2 Write Tests for Any Admin Extensions You Create 

For our purposes, weVe found that while clients enjoy the more modern themes, you have to be careful 
of howfar you extend these admin skins. What works great in vanilla d j ango. contri b. admi n can 
breakin a custom skin. Since the custom skins have to wrap portions of d j ango . contri b. admi n 
abstractions in curious ways, debugging these problems can prove to be a mind-numbing nightmare. 

Therefore, if you use a custom skin, the best practice is to write tests of the admin, especially for any 
customization. Yes, it is a bit of work up front, but it means catching these bugs much, much earlier. 

For more on testing, see our writings on testing in Chapter 22: Testing Stinks and Is a Waste of 
Money!. 


19.8 Secure the Django Admin 

Since the Django admin gives your site admins special powers that ordinary users don’t have, it’s good 
practice to make it extra secure. 


19.8.1 Change the Default Admin URL 

By default, the admin URL is yoursite.com/admin/. Change it to something thats long and difhcult 
to guess. 


TIP: Jacob Kaplan-Moss Talks About Changing the Admin URL 


Django project co-leader Jacob Kaplan-Moss says (paraphrased) that it’s an easy additional 
layer of security to come up with a different name (or even different domain) for the admin. 

It also prevents attackers from easily profiling your site. For example, attackers can teli which 
version of Django youre using, sometimes down to the point-release level, by examining the 
content of admin/ on a project. 
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19.8.2 Use django-admin-honeypot 

If youre particularly concerned about people trying to break into your Django site, django-admin- 
honeypot is a package that puts a fake Django admin login screen at admin/ and logs information 
about anyone who attempts to log in. 

See gi thub. com/dmpayton/d j ango-admi n-honeypot for more information. 


19.8.3 OnlyAUowAdmin Access via HTTPS 

This is already implied in Section 26.6: HTTPS Everywhere, but we want to especially emphasize 
here that your admin needs to be TLS-secured. If your site allows straight HTTP access, you will 
need to run the admin on a properly-secured domain, adding to the complexity of your deployment. 
Not only will you need a second deployment procedure, but you’ll need to include logic in your 
URLConf in order to remove the admin from HTTP access. In the experience of the authors, it’s 
much easier to put the whole site on TLS/HTTPS. 

Without TLS, if you log into your Django admin on an open WiFi network, it’s trivial for someone 
to sniff your admin username/password. 


19.8.4 Limit Admin Access Based on IP 

Configure your web server to only allow access to the Django admin to certain IP addresses. Look 
up the instructions for your particular web server. 

>- Nginxinstructions tech.marksblogg.com/django-admin-logins.html 

An acceptable alternative is to put this logic into middleware. It’s better to do it at the web server 
level because every middleware component adds an extra layer of logic wrapping your views, but in 
some cases this can be your only option. For example, your platform-as-a-service might not give you 
fine-grain control over web server configuration. 
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19.9 Securing the Admin Docs 

Since the Django admin docs give your site admins a view into how the project is constructed, it’s 
good practice to keep them extra-secure just like the Django admin. Borrowing from the previous 
section on the Django admin, we advocate the following: 

>- Changing the admin docs URL to something htsiAts yoursite.com/admin/doc/. 

>■ Only allowing admin docs access via HTTPS. 

>- Limiting admin docs access based on IR 

19.10 Summary 

In this chapter we covered the following: 

>- Who should be using the Django admin. 

>- When to use the Django admin and when to roU a new dashboard. 

>- String representation of objects. 

>- Adding callables to Django admin classes. 

>- Using Djangos admin docs. 

>- Encouraging you to secure the Django admin. 

>- Advised on working with custom Django skins. 
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20 Dealing With the User Model 


Django comes with a built-in support for user records. It’s a useful feature, doubly so once you learn 
how to extend and expand on the basic functionaHty. So let’s go over best practices for Django 1.11. 


20.1 Use Django’s Tools for Finding the User Model 


The advised way to get to the user class is as follows: 



It is possible to get two different User model definitions depending on the project configuration. 
This doesht mean that a project can have two different User models; it means that every project can 
customize its own User model. 
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20.1.1 Use settings.AUTH_USER_MODEL for Foreign Keys to User 

In Django 1.11, the official preferred way to attach ForeignKey, OneToOneField, or Many- 
ToManyField to User is as foUows: 


Example 20.2: Using settings.AUTH_USER_MODEL to Define Model Relations 


from django.conf import settings 
from django.db import models 

class IceCreamStore (models.Model): 

owner = models.OneToOneField(settings.AUTH_USER_MODEL) 
title = models.CharField(max_length=255) 


Yes, it looks a bit strange, but thats what the official Django docs advise. 



Figure 20.1: This looks strange too. 


WARNING: Don’t Change settings.AUTH_USER_MODEL! 


Once set in a project, changing setti ngs. AUTH_USER_MODEL requires changing your 
database schema accordingly. It’s one thing to add or modify User model fields, it’s another 
thing to create a whole new User object. 


20.1.2 Don’t Use get_user_inodel() for Foreign Keys to User 


This is bad, as it tends to create import loops. 
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Example 20.3: Using get_user_model() Improperly 
# D0N'T DO THIS! 

from django.contrib.auth import get_user_model 
from django.db import models 

class IceCreamStore(models.Model): 

# This following line tends to create import loops. 
owner = models.OneToOneField(get_user_model()) 
title = models.CharField(max_length=255) 


20.2 Custom User Fields for Django 1.11 Projects 


In Django 1.11, as long as we incorporate the required methods and attributas, we can create our 
own User model with its own fields. 


PACKAGE TIP: Libraries for Defining Custom User Models 


django-authtools is a library that makes defining custom user models easier. Of particular 
use are the AbstractEmai lUser and AbstractNamedUser models. Even ifyou don’t end 
up using django-authtools, the source code is weU worth examining. 


20.2.1 Option 1: Subclass AbstractUser 


Choose this option if you like Djangos User model fields the way they are, but need extra fields. Eor 
what it’s worth, this is the first approach that we look at anytime we start a new project. When using 
django-authtools’ base models, forms, and admin objects, we find that it’s the quickest and easiest 
way to implement custom user models. 


Here’s an example of how to subclass AbstractUser: 
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The other thing you have to do is set this in your settings: 



20.2.2 Option 2: Subclass AbstractBaseUser 

AbstractBaseUser is the bare-bones option with only 3 fields: password, last_login, and 
is_active. 

Choose this option if: 

>- Youre unhappy with the fields that the User model provides by default, such as fi rst_name 
and last_name. 

>- You prefer to subclass from an extremely bare-bones clean slate but want to take advantage of 
the AbstractBaseUser sane default approach to storing passwords. 

If you want to go down this path, we recommend the foUowing reading: 

Official Django Documentation Example 

docs.dj angoproj ect.com/en/1.11/topics/auth/customizing/ 

#a-full-example 

Source code ofdjango-authtools (EspeciaUy admi n. py, forms. py, and models. py) 
github.com/fusionbox/dj ango-authtools 
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20.2.3 Option 3: Linking Back From a Related Model 

This code is very similar to the pre-Django 1.5 project technique of creating ‘Profile’ models. Before 
discarding this approach as legacy, considet the following use cases: 

Use Case: Creating a Third Party Package 

>■ We are creating a third-party package for puhlication on PyPI. 

>- The package needs to store additional information per user, perhaps a Stripe ID or another 
payment gateway identifier. 

>- We want to be as unobtrusive to the existing project code as possible. Loose coupling! 

Use Case: Internal Project Needs 

>■ We are working on our own Django project. 

>- We want different types of users to have different fields. 

>- We might have some users with a combination of different user types. 

>- We want to handle this at the model level, instead of at other levels. 

>- We want this to be used in conjunction with a custom user model from options #1 or #2. 

Either of these use cases provide motive for the continued use of this technique. 

To make this technique work, we continue to use dj ango. contri b. models. User (caUed prefer- 
ably via dj ango. contri b. auth. get_user_model ()) and keep your related fields in separate 
models (e.g. Profiles). Here’s an example: 
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# If you do this you need to either have a post_save signal or 

# redirect to a profile_edit view on initial login. 
user = models.OneToOneField(settings.AUTH_USER_MODEL) 

favorite_ice_cream = models.ForeignKey(Elavor, null=True, blank=True) 

class ScooperProfile (models.Model) : 

user = models . OneToOneFi eld (setti ngs . AUTFI_USER_MODEL) 
scoops_scooped = models.IntegerField(default=0) 

class InventorProfile(models .Model): 

user = models . OneToOneFi eld (setti ngs . AUTFI_USER_MODEL) 

flavors_invented = models.ManyToManyField(Flavor, null=True, blank=True) 


Using this approach, we can query for any users favorite ice cream trivially with the ORM: 
user. eat e rpro fi le. favori te_i ce_cream. In addition, Scooper and Inventor profiles pro¬ 
vide individual data that only applies to those users. Since that data is isolated into dedicated models, 
it’s much harder for accidents between user types to occur. 

The only downside to this approach is that it’s possible to take it too far in complexity of profiles or 
in the supporting code. As always, keep your code as simple and clean as possible. 


WARNING: Third-Party Libraries Should Not Be Defining the User 
Model 


Unless the express purpose of the library is to define custom user models for a project (a la 
django-authtools), third-party libraries shouldht be using options #1 or #2 to add fields to 
user models. Instead, they should rely on option #3. 


20.3 Summary 

In this chapter we covered the new method to find the User model and define our own custom ones. 
Depending on the needs of a project, they can either continue with the current way of doing things 
or customize the actual user model. 
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20.3: Summary 


The next chapter is a dive into the world of third-party packages. 
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21 


Django s Secret Sauce: 
Third-Party Packages 


The real power of Django is more than just the framework and documentation available at 
dj angoproj ect. com. It’s the vast, growing selection of third-party Django and Python packages 
provided by the open source community. There are many, many third-party packages available for 
your Django projects which can do an incredible amount ofwork for you. These packages have been 
written by people from all walks of life, and they power much of the world today. 



Figure 21.1: A jar of Djangos mysterious secret sauce. Most don’t have a clue what this is. 


Much of professional Django and Python development is about the incorporation of third-party 
packages into Django projects. If you try to write every single tool that you need from scratch, youTl 
have a hard time getting things done. 


This is especially true for us in the Consulting world, where client projects consist of many of the 
same or similar building blocks. 
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21.1 Examples of Third-Party Packages 

Appendix A: Packages Mentioned In This Book covers ali of the packages mentioned throughout 
Two Scoops of Django. This list is a great starting point if youre looking for highly-useful packages 
to consider adding to your projects. 

Note that not ali of those packages are Django-specific, which means that you can use some of them 
in other Python projects. (Generally, Django-specific packages generaUy have names prefixed with 
“django-” or “dj-”, but there are many exceptions.) 

21.2 Know About the Python Package Index 

The Python Package Index (PyPI), located at pypi . python. org/pypi , is a repository of Software 
for the Python programming language. As of the time this sentence was written, it lists over 100,000 
packages, including Django itself 

For the vast majority of Python community, no open source project release is considered official until 
it occurs on the Python Package Index. 

The Python Package Index is much more than just a directory. Think of it as the worlds largest center 
for Python package information and files. Whenever you use pip to instaU a particular release of 
Django, pip downloads the files from the Python Package Index. Most Python and Django packages 
are downloadable from the Python Package Index in addition to pip. 
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21.3 Know About DjangoPackages.org 

Django Packages (d j angopackages. org) is a directory of reusable apps, sites, tools and more for 
your Django projects. Unlike PyPI, it doesnt store the packages themselves, instead providing a mix 
of hard metrics gathered from the Python Package Index, GitHuh, ReadTheDocs, and “soft” data 
entered by users. 

Django Packages is best known as a comparison site for evaluating package features. On Django 
Packages, packages are organized into handy grids so they can be compared against each other. 

Django Packages also happens to have been created by the authors of this book, with contribu- 
tions from many, many people in the Python community. Thank to the current maintainer, Jannis 
Gebaueur, and others it is continually maintained and improved as a helpful resource for Django 
users. 


21.4 Know Your Resources 

Django developers unaware of the critical resources of Django Packages and the Python Package 
Index are denying themselves one of the most important advantages of using Django and Python. If 
you are not aware of these tools, it’s well worth the time you spend educating yourself 

As a Django (and Python) developer, make it your mission to use third-party libraries instead of 
reinventing the wheel whenever possible. The best libraries have been written, documented, and 
tested by amazingly competent developers working around the world. Standing on the shoulders of 
these giants is the difference between amazing success and tragic downfaU. 

As you use various packages, study and learn from their code. You’Il learn patterns and tricks that 
wiU make you a better developer. 

On the other hand, it’s very important to be able to identify the good packages from the bad. It’s 
well worth taking the time to evaluate packages written by others the same way we evaluate our own 
Work. We cover this later in this chapter in Section 21.10: What Makes a Good Django Package? 

21.5 Tools for Installing and Managing Packages 

To take full advantage of all the packages available for your projects, having virtualenv and pip 
installed isnt something you can skip over. It’s mandatory. 
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Refer to Chapter 2: The Optimal Django Environment Setup for more details. 

21.6 Package Requirements 

As we mentioned earlier in Chapter 5: Settings and Requirements Files, we manage our Djan- 
go/Python dependencies with requirements files. These files go into the requirements/ directory that 
exists in the root of our projects. 

21.7 Wiring Up Django Packages: The Basies 

When you find a third-party package that you want to use, follow these steps: 


21.7.1 Step 1: Read the Documentation for the Package 

Are you sure you want to use it? Make sure you know what youre getting into hefore you install any 
package. 


21.7.2 Step 2: Add Package and Version Number to Your 
Requirements 

Ifyou recall from Chapter 5: Settings and Requirements Files, a requirements/base.txt file looks some- 
thing like this (but probably longer): 



Note that each package is pinned to a specific version number. Always pin your package dependencies 
to version numbers. 
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What happens if you don’t pin your dependencies? You are almost guaranteed to run into problems at 
some point when you try to reinstall or change your Django project. When new versions of packages 
are released, you eant expect them to be backwards-compatible. 

Our sad example'. Once we foUowed a software-as-a-service platform’s instructions for using their 
library. As they didnt have their own Python client, but an early adopter had a working implemen- 
tation on GitHub, those instructions told us to put the following into our requirements/base.txt: 


Example 21.2: How Not To List Requirements 

-e g"it+https://g"ithub.com/erly-adptr/py-junk.git#egg=py-jnk 


Our mistake. We should have known better and pinned it to a particular git revision number. 

Not the early adopter’s fault at aU, but they pushed up a broken commit to their repo. Once we had 
to fix a problem on a site very quickly, so we wrote a bug fix and tested it locally in development. It 
passed the tests. Then we deployed it to production in a process that grabs ali dependency changes; 
of course the broken commit was interpreted as a valid change. Which meant, while fixing one bug, 
we crashed the site. 

Not a fun day. 

The purpose of using pinned releases is to add a little formality and process to our published work. 
Especially in Python, GitEIub and other repos are a place for developers to publish their work-in- 
progress, not the final, stable work upon which our production-quality projects depend. 

One more thing, when pinning dependencies, try to pin the dependencies of dependencies. It just 
makes deployment and testing that much more predictable. 


21.7.3 Step 3: Install the Requirements Into Your Virtualenv 

Assuming you are already in a working virtualenv and are at the <repo_root> of your project, you 
pi p 1 nstall the appropriate requirements file for your setup, e.g. requirements/dev.txt. 

If this is the first time you ve done this for a particular virtualenv, it’s going to take a while for it to 
grab aU the dependencies and install them. 
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21.7.4 Step 4: Follow the Package’s Installation Instructions 
Exactly 

Resist the temptation to skip steps unless youre very familiar with the package. Since open source 
Django package developers tend to take pride in their documentation and love to get people to use 
their packages, most of the time the installation instructions theyVe authored make it easy to get 
things running. 

21.8 Troubleshooting Third-Party Packages 

Sometimes you run into prohlems setting up a package. What should you do? 

First, make a serious effort to determine and solve the prohlem yourself. Pore over the documentation 
and make sure you didht miss a step. Search online to see if others have run into the same issue. Be 
wiUing to roll up your sleeves and look at the package source code, as you may have found a bug. 

If it appears to be a bug, see if someone has already reported it in the package repository’s issue tracker. 
Sometimes youU find workarounds and fixes there. If it’s a bug that no one has reported, go ahead 
and file it. 

If you stiU get stuck, try asking for help in all the usual places: StackOverflow, IRC #django, the 
projects IRC channel if it has its own one, and your local Python user group. Be as descriptive and 
provide as much context as possible about your issue. 

21.9 Releasing Your Own Django Packages 

Whenever you write a particularly useful Django app, consider packaging it up for reuse in other 
projects. 

The best way to get started is to foUow Django’s Advanced Tutorial: How to Write Reusable Apps, for 
the basies: docs. dj angoproj ect. com/en/1 . ll/i ntro/reusable-apps/ 

In addition to what is described in that tutorial, we recommend that you also: 

>- Create a public repo containing the code. Most Django packages are hosted on GitHub these 
days, so it’s easiest to attract contributors there, but various alternatives exist (Gitlab, Bitbucket, 
Assembla, etc.). 
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>■ Release the package on the Python Package Index (pypi.python.org). Fol- 
low the submission instructions at packaging.python.org/distributing/ 
#uploading-your-proj ect-to-pypi . 

>- Add the package to Django Packages (dj angopackages. org). 

>■ Use Read the Docs (readthedocs. i o) to host your Sphinx documentation. 


TIP: Where Should I Create a Public Repo? 


There are websites that oIFer free source code hosting and version control for open source 
projects. As mentioned in Chapter 2: The Optimal Django Environment Setup, GitHub or 
GitLab are two popular options. 

When choosing a hosted version control Service, keep in mind that pip only supports Git, 
Mercurial, Bazaar, and Subversion. 


21.10 What Makes a Good Django Package? 

Here’s a checklist for you to use when creating a new open source Django package. Much of this 
applies to Python packages that are not Django-specific. This checklist is also helpful for when youre 
evaluating a Django package to use in any of your projects. 


21.10.1 Purpose 

Your package should do something useful and do it well. The name should be descriptive. The package 
repos root folder should be prefixed with ‘django-’ or ‘dj-’ to help make it easier to find. 

If part of the package’s purpose can be accomplished with a related Python package that doesht 
depend on Django, then create a separate Python package and use it as a dependency. 


21.10.2 Scope 

Your package’s scope should be tightly focused on one small task. This means that your application 
logic wiU be tighter, and users wiU have an easier time patching or replacing the package. 


283 








Chapter 21: Django’s Secret Sauce: Third-Party Packages 


21.10.3 Documen tation 

A package without documentation is a pre-alpha package. Docstrings don’t suffice as documentation. 

As described in Chapter 23: Documentation: Be Obsessed, your docs should be written in ReStruc- 
turedText. A nicely-formatted version of your docs should be generated with Sphinx and hosted 
publicly. We encourage you to use readthedocs. i o with webhooks so that your formatted docu¬ 
mentation automatically updates whenever you make a change. 

If your package has dependencies, they should be documented. Your packages installation instruc- 
tions should also be documented. The installation steps should be bulletproof. 


21.10.4 Tests 


Your package should have tests. Tests improve reliability, make it easier to advance Python/Django 
versions, and make it easier for others to contribute effectively. Write up instructions on how to run 
your packages test suite. If you or any contributor can run your tests easily before submitting a pull 
request, then youre more likely to get better quality contributions. 


21.10.5 Templates 


In the past, some Django packages provided instructions for creating templates in their docs in lieu of 
actual template files. However, nowadays it’s pretty Standard for Django packages to come with a set 
of barebones templates that demonstrate basic functionality. TypicaUy these templates contain only 
minimalist HTML, any needed JavaScript, and no CSS. The exception is for packages containing 
widgets that require CSS styling. 


21.10.6 Activity 

Your package should receive regular updates from you or contributors if/when needed. When you 
update the code in your repo, you should consider uploading a minor or major release to the Python 
Package Index. 
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21.10.7 Community 

Great open source packages, including those for Django, often end up receiving contributions from 
other developers in the open source community. All contributors should receive attribution in a 
CONTRIBUTORS.rst or AUTHORS.rst file. 

Be an active community leader if you have contributors or forks of your package. If your package is 
forked by other developers, pay attention to their work. Consider if there are ways that parts or all 
of their work can be merged into your fork. If the packages fimctionality diverges a lot from your 
package’s purpose, be humble and consider asking the other developer to give their fork a new name. 


21.10.8 Modularity 

Your package should be as easily pluggable into any Django project that doesnt replace core compo- 
nents (templates, ORM, etc) with alternatives. Installation should be minimaUy invasive. Be careful 
not to confuse modularity with over-engineering, though. 


21.10.9 Availability on PyPI 

All major and minor releases of your package should be available for download from the Python 
Package Index. Developers who wish to use your package should not have to go to your repo to get 
a working version of it. Use proper version numbers per the next section. 


21.10.10 Uses the Broadest Requirements Specifiers Possible 


Your third-party package should specify in setup.py the i nstall_requi res argument what other 
libraries your package requires in the broadest terms possible. However, this is a terrible way to define 
a package’s requirements: 
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The reason is dependency graphs. Every so often something that you absolutely pin to a specific 
version of Django or another library will break on someone else’s site project. For example, what 
if icecreamratings.com was our site and this was its deployed project’s requirements.txt file, and we 
installed django-blarg? 


Example 21.4: Requirements.txt for icecreamratings.com 


# requirements.txt for the mythical web site 'icecreamratings.com' 
Dj ango==l.11 

requests==2.13.0 
dj ango-blarg==l.0 

# Note that unlike the django-blarg library , we explicitly pin 

# the requirements so we have total control over the environment 


What would happen ifBad Example 21.2 were installed to a project with Example 21.3 requirements 
is that the Django 1.10.2 requirement would overwrite the Django 1.11 specification during installa- 
tion of icecreamratings.com requirements. As there are several backwards incompatibilities between 
Django 1.10.2 and 1.11, django-blarg could make icecreamratings.com site simply throw HTTP 500 
errors. 

Your third-party package should specify what other libraries your package requires in the broadest 
terms possible: 


Example 21.5: Broadly Defined Package Requirements 


# requirements for django-blarg 

Dj ango> = l.10,<1.12 
requests>=2.6.0,<=2.13.0 
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Additional Reading: 

>- pip.pypa.io/en/stable/reference/pip_instali/#requirement-specifiers 
>• nvie.com/posts/pin-your-packages/ 


21.10.11 ProperVersionNumbers 

Like Django and Python, we prefer to adhere to the striet version of PEP 386 naming schema. In 

fact we follow the ‘A.B.C’ pattern. Let’s go through each element: 

A’ represents the major version number. Increments should only happen with large changes that 
break backwards compatibility from the previous major version. It’s not uncommon to see 
large API changes between versions. 

‘B’ is the minor version number. Increments include less breaking changes, or deprecation notices 
about fortheoming changes. 

‘C’ represents bug fix releases, and purists call this the ‘micro’ release. It’s not uncommon for devel- 
opers to wait until a project has its first release at this level before trying the latest major or 
minor release of an existing project. 

For alpha, beta, or release-candidates for a project, the convention is to place this information as a 

suffix to the upcoming version number. So you might have: 

>- Django 1.11-alpha 
>- django-crispy-forms 1.6.1-beta 


WARNING: Don’t Upload Unfinished Code to PyPI 


PyPI, the Python Package Index, is meant to be the place where dependable, stable packages 
can be harnessed to build Python projects. PyPI is not the place for Alpha, Beta, or Release 
Candidate code, especially as pip and other tools wiU feteh the latest release by default. 

Be nice to other developers and follow the convention of only placing proper releases on PyPI. 

Note: While recent versions of pip no longer install pre-releases by default, it’s dangerous to 
expect users of code to have the latest pip version installed. 
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Additional Reading: 

>- python.org/dev/peps/pep-0386 

>- semver.org 

21.10.12 Name 

The name of the project is absolutely critical. A well-named project makes it easy to discover and 
remember, a poor name hides it from potential users, can scare ofFits use from some developer shops, 
and even block it from being listed on PyPI, Django Packages, and other resources. 

We did cover the basies in Section 4.2: What to Name Your Django Apps, but here are tips that 
apply to open source Django packages: 

>- Check to see that the name isnt already registered on PyPI. Otherwise, it won’t be trivial to install 
with pip. 

>- Check to see that the name isnt on Django Packages. This applies only to packages designed for 
use with Django. 

>■ Dont use names that include obscenity. While you might find them funny, it’s unfortunate for 
others. For example a noted developer once created a library that couldnt be used at NASA 
until he agreed to change the name. 


21.10.13 License 

Your package needs a license. Preferably, for individuals it should be licensed under the MIT licenses, 
which are generally accepted for being permissive enough for most commercial or noncommercial 
uses. If you are worried about patents, then go with the Apache license. 

Create a LICENSE.rst file in your repo root, mention the license name at the top, and paste in the 
appropriate text from the (OSI) approved list at chooseali cense. com. 


TIP: Licenses Protect You and the World 


In this era of casual litigation and patent troUs adding a Software license isnt just a matter of 
protecting your ownership of the code. It’s much, much more. If you don’t license your code, 
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or use an unapproved license not vetted by real lawyers, you run the risk of your work being 
used as a weapon by a patent troll, or in the case of financial or medical disaster, you could 
be held liable. 

OSI-approved licenses ali include a couple critical statements on Copyright, redistribution, 
disclaimer of warranty, and limitation of liabdity. 


21.10.14 Clarity of Code 

The code in your Django package should be as ciear and simple as possible, of course. Don’t use weird, 
unusual Python/Django hacks without explaining what you are doing. 


21.10.15 Use URL Namespaces 

Described in Section 8.4: Use URL Namespaces, URL namespaces allow for greater interoperability. 
Using means it’s easier to manage coUisions between projects, or even prepare for it ahead of time. 

If there is concern about future coUisions, settings-based URL namespace systems can be imple- 
mented. This is where the project defines its URL namespace as a setting, then provides a Django 
context processor and detaUed instructions on use. While it’s not hard to implement, it does create a 
level of abstraction that can make a project a little bit harder to maintain. 

21.11 Creating Your Own Packages the Easy Way 

Releasing your own bit of code can be a wonderfuUy rewarding experience. Everyone should do it! 

That said, putting all the pieces together in order to make a reusable Django package is a lot of work, 
and it’s common to get things wrong. Fortunately, Cookiecutter makes this easy. 


PACKAGE TIP: Cookiecutter: Project Templates Made Easy 


In 2013 Audrey created the popular Cookiecutter utility for generating project templates. It’s 
easy to use and very powerful. Numerous templates exist for Python and Django packages. 
Even better, many IDEs such as PyCharm and Visual Studio Code now provide support for 
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Cookiecutter-based templates. 

>- github.com/audreyr/cookiecutter 
>- cookiecutter.readthedocs .10 


In the Cookiecutter templates referenced below, we have vetted them by aggressively asking for them 
to be reviewed by leaders in both the Django and Python communities. Just use the foUowing bash 
example at the command-line: 



You’ll be prompted to provide information. The generated resuit will be an implementation of a base 
Django/Python/etc. package template that includes code, documentation, tests, license, and much 
more. 


21.12 Maintaining Your Open Source Package 


WARNING: Open Source Burnout and GivingToo Much 


Unless you are getting paid professionally to do open source work, remember that this is 
volunteer work done for pleasure. Do what you can at your own pace, and just try your best. 


The open source packages that you create have a life of their own. They mature over time, changing 
as their needs and the development standards grow over time. Here are some things we should do 
when maintaining open source projects: 
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21.12.1 Give Credit for Pull Requests 

When someone submits a pull request thats accepted, treat them right. Make sure to add the contrib- 
utor to a project’s author document called something like CONTRlBUTORS.txt or AUTHORS.txt. 

21.12.2 Handling Bad Pull Requests 

Sometimes you get pull requests that you just have to reject. Be nice and positive about it, since a 
well-handled rejected pull request can make a friend for life. 

Here are problematic pull requests that should be considered for rejection: 

>- Any pull request that fails the tests. Ask for fixes. See Chapter 22: Testing Stinks and Is a Waste 
of Money!. 

>- Any added code that reduces test coverage. Again, see chapter 22. 

>- Pull requests should change/jix as little aspossible. Large, wide-sweeping changes in a pull request 
should be rejected, with comments to isolate changes in smaller, atomic pull requests. 

>- Overly complex code submissions should be carefully considered. There is nothing wrong with asking 
for simplification, better comments, or even rejecting an overly complex pull request. 

>- Code that breaks PEP-8 needs to be resubmitted. The Django world foUows PEP-8 very closely, 
and so should your project. Submissions that violate PEP 8 can be requested to be improved. 

>- Code changes combined with major whitespace cleanup. If someone submits a change of two lines 
of code and corrects 200 lines of whitespace issues, the diff on that puU request is functionally 
unreadable and should be rejected. Whitespace cleanups need to be in their own pull request. 


WARNING: Code Changes Combined With Major Whitespace Cleanup 


We’re adding a warning because this is arguably a form of code obfuscation by a third party. 
One could argue it’s potentially a security risk. What better way to inject malignant code 
than through a pull request? 


21.12.3 Do Formal PyPI Releases 

In the Python community, it’s considered irresponsible to force developers to rely on a ‘stable’ master 
or trunk branch of critical open source projects because the PyPI version is out of date. This can 
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cause problems as open source code repositories are not considered to be good sources of production 
quality code. For example, which particular commit or tag should be used? On the other hand, PyPI, 
is a known resource designed to securely provide valid installable packages. 

In the Python world, the accepted best practice is to release when significant (or even minor) changes 
or bug fixes happen on trunk or master. In fact, minor bug fix releases are a part of every ongoing 
Software project and no one faults anyone for these kinds of things (except in US government IT 
contracts, but thats outside the scope of this book). 

If you arent sure how this works, please look at python-requests change history, it being 
one of Pythons most popular projects: gi thub. com/kennethrei tz/requests/blob/master/ 
HISTORY.rst 

To create and upload your distribution, use the foUowing steps: 



PACKAGE TIP: What is Twine? 


Twine is the preferred library for uploading packages to PyPI. The problem with python 
Setup . py is that it sends files over a non-SSH connection, exposing your library to a man- 
in-the-middle attack. In contrast, twine uses only verified TLS to upload your package. 

That’s not all! Twine works better at uploading Wheels (see the next subsection), doesnt 
require executing the setup.py, and even pre-signs your releases. If you are seriously security 
minded, it’s the tool of choice. 


21.12.4 Create and Deploy Wheels to PyPI 

According to PEP 427, Wheels are the new Standard of python distribution. They are intended to 
replace eggs and provide a number of advantages including faster installation and allow secure digital 
signing. Support is offered in pip >= 1.4 and setuptools >= 0.8. 
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Then, afteryouVe deployed your package to PyPI, run the following commands: 



For supporting Python 2.7 and 3.3+, Twine makes universal wheels when the optional setup.cfg file 
is at the same level as setup.py and includes this snippet: 



Wheel Resources: 

Specification: PEP 427 python.org/dev/peps/pep-0427 
Wheel Package on PyPI pypi .python.org/pypi /wheel 
Documentation wheel. readthedocs. i o 
Advocacy pythonwheels.com 


21.12.5 Upgrade the Package to New Versions of Django 


Every once in awhile, Django is updated with a minor release. Approximately once a year there is 
a major Django release. When this happens, it’s very important to run our package’s test suite in a 
virtualenv that contain Djangos latest release. 

If for no other reason, this is an excellent reason to include tests in your project. 
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21.12.6 Follow Good Security Practices 


We discuss security in-depth in Chapter 26: Security Best Practices. However, core Django, Python, 
and PyPy developer 7\lex Gaynor has an incredibly useful article for maintainers of any open source 
project: 

alexgaynor.net/2013/oct/19/security-process-open-source-proj ects 


TIP: Alex Gaynor on Security for Open Source Projects 


“Security vulnerabilities put your users, and often, in turn, their users at risk. As 
an author and distributor of Software, you have a responsibility to your users to 
handle security releases in a way most likely to help them avoid being exploited.” 


21.12.7 Provide Sample Base Templates 


Always include some basic templates for views using your project. We prefer to write either incredibly 
simple HTML or use a common front-end frameworks such as Twitter Bootstrap. This makes ‘test- 
driving’ the project much easier for developers who are considering using it to solve their problems. 
Invariably they’ll modify the templates inside their own templates/ directory, but this just makes 
everything so much easier. 

In addition, include a templates/myapp/base.html to increase interoperability. You can see a descrip- 
tion and example of this in cookiecutter-djangopackage: 

bit.ly/2onSzCV 


21.12.8 Give the Package Away 


Sometimes, life takes you away from maintaining a package. It might be family or a new job, but 
sometimes you just have no need for a particular open source project. Time considerations might 
mean that you don’t have the ability to reviewpull requests or explore ideas for newfeatures. Ifyoure 
the creator of a project it can be extremely chaUenging to let it go. 
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However, by giving a project away to an active maintainer, it can be reborn and prove more useful. It 
also earns the respect of the developer community at large. 

Some notable giveaways in the Django and Python communities include: 

>- lan Bicking and pip/virtualenv. 

>• Daniel and Audrey Roy Greenfeld and djangopackages.org 
>■ Daniel Roy Greenfeld and django-uni-form and dj-stripe. 

>• Rob Hudson and django-debug-toolbar. 

21.13 Additional Reading 

The following are links to useful articles for anyone contributing to, creating, or maintaining open 
source libraries: 

>■ d j angoappscheckli st. com A personal favorite of ours, the Django Apps Checklist is a 
checklist for everything in this chapter. 

>- alexgaynor.net/2013/sep/26/effective-code-review 

>- hynek.me/articles/shari ng-your-labor-of-love-pypi-quick-and-di rty 

>- j effknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the- right-way 

21.14 Summary 

Djangos real power is in the vast selection of third-party packages available to you for use in your 
Django projects. 

Make sure that you have pip and virtualenv installed and know how to use them, since theyre your 
best tools for instaUing packages on your system in a manageable way. 

Get to know the packages that exist. The Python Package Index and Django Packages are a great 
starting point for finding Information about packages. 

Package maturity, documentation, tests, and code quality are good starting criteria when evaluating 
a Django package. 

Installation of stable packages is the foundation of Django projects big and small. Being able to 
use packages means sticking to specific releases, not just the trunk or master of a project. Barring a 


295 




Chapter 21: Django’s Secret Sauce: Third-Party Packages 


specific release, you can rely on a particular commit. Fixing problems that a package has with your 
project takes diligence and time, but remember to ask for help if you get stuck. 

We also covered how to create your own third-party package, and provided basic instruction on how 
to use cookiecutter to jump-start you on your way to releasing something on the Python Package 
Index. We also included instructions on using the new Wheel format. 

Finally, we provided guidance on how to maintain a package. 
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There, got you to this chapter. 

Now you have to read it. 

WeTl try and make this chapter interesting. 

22.1 Testing Saves Money, Jobs, and Lives 

DanieVs Story. Ever hear the term “smoke test”? 

Gretchen Davidian, a Management and Program Analyst at NASA, told me that when she was stili 
an engineer, her job as a tester was to put equipment intended to get into space through such rigorous 
conditions that they would begin emitting smoke and eventually catch on fire. 

That sounds exciting! Employment, money, and lives were on the line, and knowing Gretchens at- 
tention to detail. Em sure she set a lot of hardware on fire. 

Keep in mind that for a lot of us, as Software engineers, the same risks are on the line as NASA. I 
recall in 2004 while working for a private company how a single miles-vs-kilometers mistake cost a 
company hundreds of thousands of dollars in a matter of hours. Quality Assurance (QA) staff lost 
their jobs, which meant that money and health benefits were gone. In other words, employment, 
money, and possibly lives can be lost without adequate tests. While the QA staff were very dedicated, 
everything was done via manually clicking through projects, and human error simply crept into the 
testing process. 
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Today, as Django moves into a wider and wider set of applications, the need for automated testing is 
just as important as it was for Gretchen at NASA and for the poor QA staff in 2004. Here are some 
cases where Django is used today that have similar quality requirements: 

>- Your application handles medical information. 

>- Your application provides life-critical resources to people in need. 

>- Your application works with other peoples money now or wiU at some point in the future. 


PACKAGE TIP: Useful Library for Testing Django Projects 


We like to use coverage.py. 

This tool provides ciear insight into what parts of your code hase are covered hy tests, and 
what lines havent been touched by tests. You also get a handy percentage of how much of your 
code is covered by tests. Even 100% test coverage doesnt guarantee a bug-free application, 
but it helps. 

We want to thank Ned Batchelder for his incredible work in maintaining coverage.py. It’s a 
superb project and is useful for any Python related project. 


22.2 How to Structure Tests 

Let’s say we ve just created a new Django app. The first thing we do is delete the default but useless 
tests.py modxsle. that python manage. py startapp creates. Init’s place we create a directory called 
tests and place an empty_ init _ .py within. 

Inside that new directory, because most apps need them, we create test_forms.py, test_models.py, 
testjviews.py modules. Tests that apply to forms go into test_forms.py, model tests go into 
test_models.py, and so on. 

Here’s what it looks like: 
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forms.py 
models.py 
tests/ 

_init_.py 

test_forms.py 
test_models.py 
test_views.py 
views.py 


Also, if we have other files besides forms.py, models.py and views.py that need testing, we create 
corresponding test files. 

We like this approach because while it does add an extra layer of nesting, the alternative is to have an 
app structure with a painful number of modules to navigate. 


TIP: Prefix Test Modules With test 


It’s criticaUy important that we always prefix test modules with test_, otherwise Django’s 
test runner eant trivially discover our test files. Also, it’s a nice convention that aUows for 
greater flexibility for viewing filenames in IDEs and text editors. As always, never lean on 
your preferred IDE for how you name things. 


22.3 How to Write Unit Tests 

It’s not uncommon for programmers to feel at the top of their game at the moment they are writing 
code. When they revisit that same code in months, weelcs, days, or even hours and it’s not uncommon 
for programmers to feel as if that same code is of poor quality. 

The same applies to writing unit tests. 

Over the years, weVe evolved a number of practices we like to follow when writing tests, including 
unit tests. Our goal is always to write the most meaningful tests in the shortest amount of time. 
Hence the foUowing: 
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22.3.1 Each Test Method Tests One Thing 

A test method must be extremely narrow in what it tests. A single unit test should never assert the 
behavior of multiple views, models, forms, or even multiple methods within a class. Instead, a single 
test should assert the behavior of a single view, model, form, method or function. 

Of course, therein lies a conundrum. How does one run a test for a view, when views often require 
the use of models, forms, methods, and functions? 

The trick is to be absolutely minimalistic when constructing the environment for a particular test, as 
shown in the example below: 



In this test, taken from code testing the API we presented in Section 16.2: lUustrating Design Con- 
cepts With a Simple API, we use the setUp () method to create the minimum possible number of 
records needed to run the test. 
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Here’s a much larger example, one based on the REST API example that we provided in Chapter 16: 
Building REST APIs With Django REST Framework. 


Example 22.3: Testing API Code 


# flavors/tests/test_api.py 

•import json 

from django.test import TestCase 
from django.uris import reverse 

from flavors.models import Flavor 

class DjangoRestFrameworkTests(TestCase) : 

def setUp(self) : 

Flavor.objects.get_or_create(title= 'titlel' , slug= 'slugl' ) 
Flavor.objects.get_or_create(title= 'title2' , slug= 'slug2' ) 

self .create_read_url = reverse (' flavor_rest_api' ) 
self .read_update_delete_url = \ 

reverse (' flavor_rest_api' , kwargs={ 'slug' : 'slugl'}) 

def test_list (self ): 

response = self .client.get (self .create_read_url) 

# Are both tities in the content? 
self .assertContains(response, 'titlel') 
self .assertContains(response, 'title2') 

def test_detai1 (self ): 

response = self .client.get (self .read_update_delete_url) 
data = json.loads(response.content) 

content = {'id': 1, 'title': 'titlel', 'slug': 'slugl', 

'scoops_remaining' : 0} 

self .assertEquals(data, content) 
def test_create (self ): 
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post = {'title': 'titleS', 'slug': 'slugS'} 

response = self .client.post (self .create_read_url, post) 

data = json.loads(response.content) 

self. assertEquals(response.status_code, 201) 

content = {'id': 3, 'title': 'titleS' , 'slug': 'slugS' , 

'scoops_remaining' : 0} 

self .assertEquals(data, content) 

self. assertEquals(Flavor.objects.count(), 3) 

def test_delete (self ): 

response = self .client.delete (self .read_update_delete_url) 
self. assertEquals(response.status_code, 204) 
self. assertEquals(Flavor.objects.count(), 1) 


22.3.2 For Views, When Possible Use the Request Factory 


The d j ango. test. cli ent. RequestFactory provides a way to generate a request instance that 
can be used as the first argument to any view. This provides a greater amount of isolation than the 
Standard Django test client, but it does require a little bit of extra work on the part of the test writer. 
This is because the request factory doesnt support middleware, including session and authentication. 

See docs.dj angoproj ect.com/en/1.11/topics/testing/advanced/ 

Unfortunately the documentation doesnt cover when you want to test a view wrapped with a single 
mi ddlewa re class. For example, if your view required sessions, this is how you would do it: 
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middleware . process_request(request) 
return request 

def add_middleware_to_response(request, middleware_class): 
middleware = middleware_class() 
middleware.process_response(request) 
return request 

class SavorylceCreamTest(TestCase) : 
def setUp (self ): 

# Every test needs access to the request factory. 
self.factory = RequestFactory() 

def test_cheese_flavors (self ): 

request = self . factory . get (' /cheesy/broccoli/' ) 
request.user = AnonymousUser() 

# Annotate the request object with a session 

request = add_middleware_to_request(request, SessionMiddleware) 
request.session.save() 

# process and test the request 
response = cheese_flavors(request) 
self . assertContains(response, 'bleah! ') 


22.3.3 Don’t Write Tests That Have to Be Tested 


Tests should be written as simply as possible. If the code in a test (or the code called to help run 
a test) feels complicated or abstracted, then you have a problem. In fact, we ourselves are guilty of 
writing overly complicated utility test functions that required their own tests in the past. As you can 
imagine, this made debugging the actual tests a nightmare. 
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22.3.4 Don’t RepeatYourself Doesn’t Apply to WritingTests 

The setUpO method is really useful for generating reusable data across ali test methods in a test 
class. However, sometimes we need similar but different data between test methods, which is where 
we often fall into the trap of writing fancy test Utilities. Or worse, we decide that rather than write 
20 similar tests, we can write a single method that when passed certain arguments will handle ali the 
Work for us. 

Our favorite method of handling these actions is to just dig in and write the same or similar code 
multiple times. In fact, weTl quietly admit to copy/pasting code between tests to expedite our work. 


TIP: Our Problem With the Django Testing Tutorial #5 


The official Django beginner tutorial on testing demonstrates the implementation of a cre- 
ate_questi on () utility method, which is designed to take some of the repetition out of 
the process of created questions. 

We think it’s inclusion in the tutorial is a mistake. Simple examples in the official documen- 
tation that follow poor practice provide encouragement that often play out in terrible ways. 
WeVe encountered projects inspired by this example to add deep layers of abstraction to their 
testing code, abstraction that makes it incredibly chaUenging to correct and enhance existing 
tests. 

Again, Don’t Repeat Yourself doesnt apply to tests. 


22.3.5 Don’t Rely on Fixtures 

WeVe learned over time that using fixtures is problematic. The problem is that fixtures are hard to 
maintain as a project’s data changes over time. Modifying JSON-formatted files to match your last 
migration is hard, especiaUy as it can be difficult to identify during the JSON load process where 
your JSON file(s) is either broken or a subtly inaccurate representation of the database. 

Ratber than wrestle with fixtures, weVe found it’s easier to write code that relies on the ORM. Other 
people like to use third-party packages. 
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PACKAGE TIP: Tools to Generate Test Data 


The foUowing are popular tools for test data generation: 

>- factoryboy A package that generates model test data. 

>- faker This package generates test data, but rather than a random jumble of text, it 
creates localized names, addresses, and text. It even comes with instructions on how 
to integrate it with factory boy: faker. readthedocs. io/en/master/ 
#how-to-use-with-factory-boy 
>- model mommy Another package that generates model test data. 

>- mock Not explicitly for Django, this allows you to replace parts of your system with 
mock objects. This project made its way into the Standard library as of Python 3.3. 


22.3.6 Things That Should Be Tested 

Everything! Seriously, you should test whatever you can, including: 

Views: Viewing of data, changing of data, and custom class-based view methods. 

Models: Creating/updating/deleting of models, model methods, model manager methods. 

Forms: Form methods, clean() methods, and custom fields. 

Validators: ReaUy dig in and write multiple test methods against each custom validator you write. 
Pretend you are a malignant intruder attempting to damage the data in the site. 

Signals: Since they act at a distance, signals can cause grief especially if you lack tests on them. 

Filters: Since filters are essentiaUy just functions accepting one or two arguments, writing tests for 
them should be easy. 

Template Tags: Since template tags can do anything and can even accept template context, writing 
tests often becomes much more chaUenging. This means you reaUy need to test them, since 
otherwise you may run into edge cases. 

Miscellany: Context processors, middleware, email, and anything else not covered in this list. 

Failure What happens when any of the above fail? Testing for system error is as important as testing 
for system success. 

The only things that shouldnt be tested are parts of your project that are already covered by tests in 

core Django and third-party packages. For example, a modefs fields don’t have to be tested if youre 

using Django’s Standard fields as-is. However, if youre creating a new type of field (e.g. by subclassing 
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Fi leFi eld), then you should write detailed tests for anything that could go wrong with your new 
field type. 




Figure 22.1: Test as much of your project as you can, as if it were free ice cream. 


22.3.7 Test for Failure 


Let’s say we have a view that allows users to edit their own ice cream shop reviews. The obvious 
tests involve logging in, attempting to change the review, and then checking whether theyVe actually 
changed. Test success, coverage 100%. Right? 

However, this only tests part of the scenario. What if the user isn’t logged in? What if the user is 
trying to edit someone elses review? Does the view produce an error and most importantly: is the 
object left unchanged? It has been argued that this test is even more important than the success 
scenario: a failure in the success scenario will cause inconvenience for users, but wiU be reported. A 
failure in the fail scenario wiU cause a silent security hole that could go undetected until it’s too late. 

This is only a sampling of the things that can go wrong when we doht test for what happens when 
our Systems break down. It is up to us to learn how to test for the exceptions our code may throw: 

>- docs.python.org/2/library/unittest.html#unittest.TestCase. 
assertRaises 
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22.3: How to Write Unit Tests 


>■ bit.ly/2pAxLtm PyTest assertion docs 

22.3.8 Use Mock to Keep Unit Tests From Touching the World 

Unit tests are not supposed to test things external to the flinction or method they are calling. Which 
means that during tests we should not access external APIs, receive emails or webhoolcs, or anything 
that is not part of the tested action. Alas, this causes a conundrum when you are trying to write a 
unit test for a function that interacts with an external API. 

At this point you have two choices: 

>- Choice #1: Change the unit test to be an Integration Test. 

>■ Choice #2: Use the Mock library to fake the response from the external API. 

The Mock library, created by Michael Foord, has as one of its features the capability to briefly monkey- 
patch libraries in order to make them return the exact value that we want. This way we are testing 
not the availability of the external API, but instead just the logic of our code. 

In the example displayed below, we are monkey-patching a function in a mythical Ice Cream 7\PI 
library so our test code doesht access anything external to our application. 
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# list_flavors_sorted0 calls the icecreamapi.get_flavors() 

# function. Since we've monkeypatched the function, it will always 

# return ['chocolate', 'strawberry', 'vanilla', ]. Which the. 

# list_flavors_sorted() will sort alphabetically 
flavors = list_flavors_sorted() 

self.assertEqual( 
flavors, 

['chocolate', 'strawberry', 'vanilla', ] 

) 


Now let’s demonstrate how to test the behavior of the li st_f lavors_sorted () function when 
the Ice Cream API is innaccessible. 



As an added bonus for API authors, heres how we test how code handles two different python- 
requests connection problems: 
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22.3: How to Write Unit Tests 


with self.assertRaises(CantListFlavors) : 
list_flavors_sorted() 

@mock.patch.obj ect (requests, 'get' ) 
def test_request_failure(self , get) 

"""Test if we can handle SSL problems elegantly." 
get.side_effect = requests.exception.SSLError() 

with self.assertRaises(CantListFlavors): 
list_flavors_sorted() 


22.3.9 Use Fancier Assertion Methods 

Comparing two lists (or tuples) is a very common use case. However, if the lists are al- 
lowed to have different sort orders, then we have to sort the lists to match, then run 
self.assertEqual(control_list, candidate_list) right? 

Not if we know to use unittests Li stItemsEqual () assertion method! In fact, Python and 
Djangos unittest documentation includes handy links to the very useful assertion types we get for 
free: 

>- docs.python.org/3/library/unittest.html#assert-methods 

>• docs.dj angoproj ect.com/en/1.11/topics/testing/tools/#assertions 

WeVe found the foUowing assert methods extremely useful: 

>• assertRaises 

>- Python2.7: ListItemsEqual(), Python 3+ assertCountEqual() 

>- assertDi ctEqual () 

>- assertFormError() 

>- assertContai ns () Check status 200, checks in response. content. 

>• assertHTMLEqual ( ) Amongst many things, ignores whitespace differences. 

>- assertdSONEqual () 
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22.3.10 Document the Purpose of Each Test 

Just as it is a good idea to document the purpose of a class, method, or function with docstrings, it 
is also a good idea to document the purpose the test analogs of these items. If undocumented code 
makes a project somewhat harder to maintain, undocumented test code can make a project impossible 
to test. To remedy this, a little hit of docstring can go a long way. 

If you think this is boring, well, weVe found that a good way to deal with an impossible-to-debug 
problem is to document the related tests. By the time the tests are documented, you have either 
figured out the problem or you have documented tests. Either case is a win! 

22.4 What About Integration Tests? 

Integration testing is when individual Software modules are combined and tested as a group. This is 
best done after unit tests are complete. Examples of integration tests can include: 

>- Selenium tests to confirm that an application works in the browser. 

>■ Actual testing against a third-party API instead of mocking responses. For example, Django 
Packages conducts periodic tests against GitHub and the PyPI API to ensure that its interac- 
tion with those Systems is valid. 

>- Interactingwith requestb . i n or httpbi n. org to confirm the validity of outbound requests. 

>- Using runscope. com to validate that our API is working as expected. 

Integration tests are a great way to confirm that ‘aU the pieces’ are working. We can confirm that our 
users will see what they are supposed to see and our APIs are functioning correctly. 

The downside of integration tests are: 

>- Setting up integration tests can take a lot of time. 

>- Compared to unit tests, integrations are extremely slow. Thats because instead of testing the 
smallest components, integration tests are, by definition, testing the whole System. 

>■ When errors are thrown by integration tests, uncovering the problem is harder than unit tests. 
For example, a problem affecting a single type of browser might be caused by a Unicode trans- 
formation happening at the database level. 

>- Integration tests are fragile compared to unit tests. A smaU change in a component or setting 
can break them. WeVe yet to work on a significant project where at least one person washt 
forever blocked from running them successfully. 


310 




22.5: Continuous Integration 


Even with these problems, integration tests are useful and worth considering adding to our testing 
stack. 


22.5 Continuous Integration 

For projects of any size, we recommend setting up a continuous integration (CI) server to run the 
projects test suite whenever code is committed and pushed to the project repo. See Chapter 32: 
Continuous Integration for more details. 

22.6 Who Cares? We Don’t Have Time for Tests! 

“Tests are the Programmers stone, transmutingfear into boredom5' 

-Kent Beck 

Let’s say you are confident of your coding skiU and decide to skip testing to increase your speed of 
development. Or maybe you feel lazy. It’s easy to argue that even with test generators and using tests 
instead of the shell, they can increase the time to get stuif done. 

Oh, really? 

What about when it’s time to upgrade? 

That’s when the small amount of work you did up front to add tests saves you a lot of work. 

For example, in the summer of 2010, Django 1.2 was the Standard when we started Django Packages 
(d j angopackages. org). Since then the project has stayed current with new Django versions, which 
has been really useful. Because of its pretty good test coverage, moving it up a version of Django (or 
the various dependencies) has been easy. Our path to upgrade: 

>■ Upgrade the version in a local instance of Django Packages. 

>- Run the tests. 

>- Fix any errors that are thrown by the tests. 

>- Do some manual checking. 

If Django Packages didnt have tests, any time we upgraded anything we would have to click through 
dozens and dozens of scenarios manually, which is error-prone. Having tests means we can make 
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changes and dependency upgrades with the confidence that our users (i.e. the Django community) 
wont have to deal with a buggy experience. 

This is the benefit of having tests. 

22.7 The Game of Test Coverage 

A great, fun game to play is trying get test coverage as high as possible. Every day that we increase 
our test coverage is a victory, and every day that the coverage goes down is a loss. 

22.8 S etting Up the Test Coverage Game 

Yes, we call test coverage a game. It’s a good tool for developers to push themselves. It’s also a nice 
metric that both developers and their clients/employers/investors can use to help evaluate the status 
of a project. 

We advocate following these steps because most of the time we want to only test our own project’s 
apps, not aU of Django and the myriad of third-party libraries that are the building bloclcs of our 
project. Testing those ‘building blocks’ takes an enormous amount of time, which is a waste because 
most are already tested or require additional setup of resources. 


22.8.1 Step 1: Start Writing Tests 


WeVe done that already, right? 


22.8.2 Step 2: Run Tests and Generate Coverage Report 

Let’s try it out! In the command-line, at the <project_root>, type: 


Example 22.8: Running Django tests using coverage.py 


$ coverage run manage.py test --settings=twoscoops.settings.test 

If we have nothing except for the default tests for two apps, we should get a response that looks like: 
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Example 22.9: Positive Test Results 


Creating test database for alias "default"... 

Ran 2 tests in 0.008s 

OK 

Destroying test database for alias "default"... 


This doesn’t look like much, but what it means is that weVe constrained our application to only run 
the tests that you want. Now it’s time to go and look at and analyze our embarrassingly low test 
coverage numbers. 


22.8.3 Step 3: Generate the Report! 


coverage.py provides a very useful method for generating HTML reports that don’t just provide 
percentage numbers of what’s been covered by tests, it also shows us the places where code is not 
tested. In the command-line, at the <project_root>\ 


Example 22.10: Test Results Without admin.py 


$ coverage html --ornit="admin.py" 


Ahem...don’t forget to change <project-root> to match the development machine’s structure! Eor ex¬ 
ample, depending on where one does things, the <path-to-project-root> could be: 

>- /Users/audreyr/code/twoscoops/twoscoops/ 

> /Users/pydanny/projects/twoscoops/twoscoops/ 

> c:\twoscoops 

After this runs, in the <project_root> directory there is a new directory caUed htmlcov/. In the htmlcov/ 
directory, open the index.html file using any browser. 
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What is seen in the browser is the test results for our test run. Unless we already wrote some tests, 
the total on the front page will be in the single digits, if not at 0%. Click into the various modules 
listed and we should see lots of code thats red-colored. Red is had. 

Let’s go ahead and admit that our project has a low coverage total. Ifyour project has a low coverage 
total, you need to admit it as well. It’s okay just so long as we also resolve to improve the coverage 
total. 

In fact, there is nothing wrong in saying publicly that you are working to improve a project’s test 
coverage. Then, other developers (including ourselves) will cheer you on! 


22.9 Playing the Game of Test Coverage 

The game has a single rule: 

Mandate that no commit can lower test coverage. 

If we add a feature or bug fix and coverage is 65% when we start, we can’t merge our code in until 
coverage is at least 65% again. At the end of each day, if test coverage goes up by any amount, it 
means we’re winning. 

Keep in mind that the gradual increase of test coverage can be a very good thing over huge jumps. 
Gradual increases can mean that we developers arent putting in bogus tests to bump up coverage 
numbers; instead, we are improving the quality of the project. 


22.10 Alternatives to unittest 

All the examples in this chapter thus far have used the unittest library. While every known authority 
on testing agrees that unittest is a very powerful, useful tool, not ah of them like it. The specific reason, 
and one we fuUy comprehend, is that it requires too much boilerplate. 

Fortunately, there are alternatives that require a lot less boilerplate. These are: 

>- pypi.python.org/pypi/pytest-dj ango/ 

>• pypi.python.org/pypi/dj ango-nose 
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22.11: Summary 


These two libraries are wrappers around the pytest and nose libraries. In return for a little bit of extra 
Setup, these libraries allow for not just the running of unittest-based tests, but also for running any 
fiinction (and class/directory/module) prefixed with “test_”. For example, you could write a simple 
test that looks like this: 



While this example is based on pytest, similar functionality can be used with nose and it’s 
nose. tools. rai ses decorator. 

A possible downside of the simplicity of these function-based tests is the lack of inheritance. If a 
project needs to have similar behavior over a lot of different tests, then writing tests this way may not 
make sense. 


22.11 Summary 

All of this might seem silly, but testing can be very serious business. In a lot of developer groups this 
subject, while gamified, is taken very seriously. Lack of stability in a project can mean the loss of 
clients, contracts, and even employment. 

In the next chapter we cover a common obsession of Python developers: documentation. 
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Documentation: Be Obsessed 


Given a choice between ice cream and writing great documentation, most Python developers would 
probably choose to write the documentation. That being said, writing documentation while eating 
ice cream is even better. 

When you have great documentation tools like reStructuredText and Sphinx, you actually can’t help 
but want to add docs to your projects. 


PACKAGE TIP: Install Sphinx Systemwide 


WeVe found that simply instaUing Sphinx fetches for us ali the pieces you need to document 
our Django (or Python) project. We recommend pip instaUing Sphinx systemwide, as you’U 
want to have it handy for every Django project. 


23.1 Use reStructuredText for Python Docs 

You’U want to learn and follow the Standard Python best practices for documentation. These days, re¬ 
StructuredText (RST) is the most common markup language used for documenting Python projects. 

What follows are linies to the formal reStructuredText specification and a couple sample projects 
which benefit from using it: 

>- docutlIs.sourceforge.net/does/ref/rst/restructuredtext.html 
>- docs.djangoproj ect.com/en/1.11/ 

>- docs.python.org 
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While it’s possible to study the formal documentation for reStructuredText and learn at least the 
basies, here is a quick primer of some very useful commands you should learn. 


Example 23.1: ReStructured Text Primer 


Section Header 

**emphas"is (bold/strong)** 

*italics* 

Simple link: https://twoscoopspress.com 
Fancier Link; 'Two Scoops of Django'_ 

.. _Two Scoops of Django; https://twoscoopspress.com 

Suhsection Header 

#) An enumerated list item 
#) Second item 

* First hullet 

* Second hullet 

* Indented Bullet 

* Note carriage return and indents 

Literal code block:: 

def like 0: 

print("I like Ice Cream") 
for i in range(10): 
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23.2: Use Sphinx to Generate Documentation From reStructuredText 


Ii ke 0 

Python colored code block (requires pygments): 
code-block:: python 

# You need to "pip install pygments" to make this work. 

for i in range(10): 
li ke 0 

lavaScript colored code block: 
code-block:: javascript 

console.log("Don't use alert()"); 

23.2 Use Sphinx to Generate Documentation From 
reStructuredText 

Sphinx is a tool for generating nice-looking docs from your .rst files. Output formats include HTML, 
LaTeX, manual pages, and plain text. 

FoUow the instructions to generate Sphinx docs: sphi nx-doc. org. 


TIP: Build Your Sphinx Documentation at Least Weekly 


You never know when bad cross-references or invalid formatting can break the Sphinx build. 
Rather than discover that the documentation is unbuildable at an awlcward moment, just 
make a habit of creating it on a regular basis. 

23.3 What Docs Should Django Projects Contain? 

Developer-facing documentation refers to notes and guides that developers need in order to set 
up and maintain a project. This includes notes on instaUation, deployment, architecture, how to run 
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tests or submit pull requests, and more. WeVe found that it really helps to place this documentation 
in all our projects, private or public. 

Here we provide a table that describes what we consider the absolute minimum documentation: 

1 Filename or Directory 

Reason 

Notes 1 

README. rst 

Every Python project you begin 
should have a README.rst file in 

the repository root. 

Provide at least a short 
paragraph describing what the 
project does. Also, link to the 
instaUation instructions in the 
docs/ directory. 

docs/ 

Your project documentation should 
go in one, consistent location. This 
is the Python community Standard. 

A simple directory. 

docs/deployment. rst 

This file lets you take a day offi 

A point-by-point set of 
instructions on how to 
instaU/update the project into 
production, even if it’s done 
via something powered by 

Ruby, Fabric, or a Makefile. 

docs/installation. rst 

This is really nice for new people 
Corning into a project or when you 
get a new laptop and need to set up 
the project. 

A point-by-point set of 
instructions on how to 
onboard yourself or another 
developer with the Software 
Setup for a project. 

docs/architecture. rst 

A guide for understanding what 
things evolved from as a project ages 
and grows in scope. 

This is how you imagine a 
project to be in simple text 
and it can be as long or short 
as you want. Good for 
keeping focused at the 
beginning of an effort. 

Table 23.1: Documentation Django Projects Should Contain 
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Figure 23.1: Even ice cream could benefit from documentation. 


23.4 Additional Documentation Resources 

>- python.org/dev/peps/pep-0257 Oflicial specification on docstrings. 

>- readthedocs. i o Read the Docs is a free Service that can host your Sphinx documentation. 
>- pythonhosted . org Python Hosted is another free Service for documentation hosting. 

23.5 The Markdown Alternative 

Markdown is a plain text formatting syntax not too dissimilar to reStructuredText. While it doesnt 
have all the built-in features of reStructuredText, it does have the advantage of being easier to learn. 
While used infrequently in the Python and Django communities, it’s very popular in tangential places 
including the JavaScript and technical book-writing community. 

When using Markdown instead of reStructuredText for open source projects, keep the following in 
mind: 

>► PyPI will not format the long_descri pti on if it’s written in anything except reStructured¬ 
Text. 
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>■ Many Python and Django developers will search reStructuredText-powered documentation 
sources before Markdown-powered ones. 


23.5.1 README.md to README.rst: Using Pandoc for Packages 
Uploaded to PyPI 


Pandoc is a command-line tool that allows us to convert files from one markup format into another. 
We can write a README in one format, and for uploading to PyPI, display it in another. Here’s 
how some people use pandoc in their setup.py module: 



To convert a README.md to README.rst, just run python setup. py md2rst. Then we up- 
load the package to PyPI as per the normal process. 


23.5.2 Markdown Resources 

>• en.wikipedia.org/wiki/Markdown 

>• mkdocs . org is a static site generator geared towards project documentation written in mark¬ 
down. 

>- documentup. com will host README documents written in Markdown format. 

>- p rogri um . vi ewdocs. i o allows for Markdown documents to be organized and displayed in 
a Sphinx-like format. Also provides free hosting. 

>• j ohnmacf arlane. net/pandoc is a useful tool for converting between Markdown to other 
formats, but it’s not perfect. It’s great for converting README.md to README.rst, which 
we describe in the previous subsection 


322 







23.6: Wikis and Other Documentation Methods 


23.6 Wikis and Other Documentation Methods 

For whatever reason, if you can’t place developer-facing documentation in the project itself, you 
should have other options. While wikis, Online document Stores, and word processing documents 
doht have the feature of being placed in version control, they are hetter than no documentation. 

Please consider creating documents within these other methods with the same names as the ones we 
suggested in the table on the previous page. 

23.7 Summary 

In this chapter we went over the following: 

>- The use of reStructuredText to write documentation in plaintext format. 

>- The use of Sphinx to render your documentation in HTML and EPUB formats. If you know 
how to instaU LaTeX you can even render it as PDF. For reference, installing LaTeX is easy 
to do on Linux and Windows and a bit harder on Mac OS X. 

>- Advice on the documentation requirements for any Django project. 

>- Using Markdown for documentation, and converting README.md to README.rst. 

Next, weTl take a look at common bottlenecks in Django projects and ways to deal with them. 
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Finding and Reducing 
Bottlenecks 


This chapter covers a few basic strategies for identifying bottlenecks and speeding up your Django 
projects. 

24.1 Should You Even Care? 

Remember, premature optimization is bad. If your site is small- or medium-sized and the pages are 
loading fine, then it’s okay to skip this chapter. 

On the other hand, if your sites user base is growing steadily or youre about to land a strategic 
partnership with a popular brand, then read on. 

24.2 Speed Up Query-Heavy Pages 

This section describes how to reduce bottlenecks caused by having too many queries, as weU as those 
caused by queries that arent as snappy as they could be. 

We also urge you to read up on database access optimization in the oflicial Django docs: docs. 
dj angoproj ect.com/en/1.11/topics/db/optimization/ 

24,2.1 Find Excessive Queries With Django Debug Toolbar 

You can use django-debug-toolbar to help you determine where most of your queries are coming 
from. You’ll find bottlenecks such as: 
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> Duplicate queries in a page. 

>- ORM calls that resolve to many more queries than you expected. 

>• Slow queries. 

You probably have a rough idea of some of the URLs to start with. For example, which pages dont 
feel snappy when they load? 

InstaU django-debug-toolbar locally if you don’t have it yet. Look at your project in a web browser, 
and expand the SQL panel. It’ll show you how many queries the current page contains. 


PACKAGE TIP: Packages for Profiling and Performance Analysis 


django-debug-toolbar is a critical development tool and an invaluable aid in page-by-page 
analysis. We also recommend adding django-cacbe-panel to your project, but only config- 
ured to run when settings/local.py module is called. This will increase visibility into what your 
cache is doing. 

django-extensions comes with a tool called RunProfi leServer that starts Djangos run- 
server command with hotshot/profiling tools enabled. 

silk (gi thub. com/mtford90/si Ik) Silicis a live profiling Django app that intercepts and 
Stores HTTP requests and database queries before presenting them in a user interface for 
further inspection. 


24,2.2 Reduce the Number of Queries 

Once you know which pages contain an undesirable number of queries, figure out ways to reduce 
that number. Some of the things you can attempt: 

>• Try using select_related () in your ORM caUs to combine queries. It follows For- 
ei gnKey relations and combines more data into a larger query. If using CBVs, django-braces 
makes doing this trivial with the SelectRelatedMi xi n. Beware of queries that get too large 
by explicitly passing the related field names you are interested in. Only the specified relations 
will be followed. Combine that with carefiil testing! 

>- For many-to-many and many-to-one relationships that eant be optimized with se- 
lect_related (), explore using prefetch_related () instead. 
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>■ If the same query is being generated more than once per template, move the query into the 
Python view, add it to the context as a variable, and point the template ORM calls at this new 
context variable. 

>- Implement caching using a key/value store such as Memcached or Redis. Then write tests to 
assert the number of queries run in a view. See 

docs.dj angoproj ect.com/en/1.11/topics/testing/tools/#dj ango.test. 

Transacti onTestCase. assertNumQueri es for instructions. 

>- Use the django.utils. functional.cached_property decorator to cache in memory 
the restdt of method caU for the life of an object instance. This is incredibly useful, so please 
see Section 29.3.5: django.utils.functional.cached_property in chapter 29. 


24,2.3 Speed Up Common Queries 


The length of time it takes for individual queries can also be a bottleneck. Here are some tips, but 
consider them just starting points: 

>- Make sure your indexes are helping speed up your most common slow queries. Look at the raw 
SQL generated by those queries, and index on the fields that you filter/sort on most frequently. 
Look at the generated WHERE and ORDER_BY clauses. 

>■ Understand what your indexes are actuaUy doing in production. Development machines will 
never perfectly replicate what happens in production, so learn how to analyze and understand 
whats reaUy happening with your database. 

>- Look at the query pians generated by common queries. 

>- Turn on your databases slow query logging feature and see if any slow queries occur frequently. 

>■ Use django-debug-toolbar in development to identify potentiaUy-slow queries defensively, be- 
fore they hit production. 

Once you have good indexes, and once youVe done enough analysis to know which queries to rewrite, 
here are some starting tips on how to go about rewriting them: 

O Rewrite your logic to return smaller resuit sets when possible. 

0 Re-model your data in a way that aUows indexes to work more effectively. 

© Drop down to raw SQL in places where it would be more efficient than the generated query. 
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TIP: Use EXPLAIN ANALYZE / EXPLAIN 


If youre using PostgreSQL, you can use EXPLAIN ANALYZE to get an extremely detailed 
query plan and analysis of any raw SQL query. For more Information, see: 

>- revsys.com/writings/postgresql-performance.html 

>- craigkerstiens.com/2013/01/10/more-on-postgres-performance/ 

The MySQL equivalent is the EXPLAIN command, which isn’t as detailed but is stiU helpful. 
For more information, see: 

>- dev.mysql.com/doc/refman/5.7/en/expiain.html 
A nice feature of django-debug-toolbar is that the SQL pane has an EXPLAIN feature. 


24,2.4 Switch ATOMI C_REQyESTS to Ealse 


The ciear, vast majority of Django projects wiU run just fine with the setting of ATOMIC_R EQUE SIS to 
True. Generally, the penalty of running all database queries in a transaction isnt noticeable. However, 
if your bottleneck analysis points to transactions causing too much delay, it’s time to change the 
project run as ATOMIC_REQUESTS to T rue. See Section 7.7.2: Explicit Transaction Declaration for 
guidelines on this setting. 

24.3 Get the Most Out ofYour Database 

You can go a bit deeper beyond optimizing database access. Qptimize the database itself! Much of 
this is database-specific and already covered in other books, so we won’t go into too much detail here. 


24,3,1 Know What Doesn’t Belong in the Database 

Frank Wiles of Revolution Systems taught us that there are two things that should never go into any 
large site’s relational database: 

Logs. Don’t add logs to the database. Logs may seem QK on the surface, especially in development. 
Yet adding this many writes to a production database wiU slow their performance. When the ability 
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to easily perform complex queries against your logs is necessary, we recommend third-party Services 
such as Splunk or Loggly, or use of document-based NoSQL databases. 

Ephemeral data. Don’t store ephemeral data in the database. What this means is data that re¬ 
quires constant rewrites is not ideal for use in relational databases. This includes examples such as 
django.contrib.sessions, django.contrib.messages, and metrics. Instead, move this data to things like 
Memcached, Redis, Riak, and other non-relational Stores. 


TIP: Frank Wiles on Binary Data in Databases 


Actually, Frank says that there are three things to never store in a database, the 
third item being binary data. Storage of binary data in databases is addressed by 
d j ango. db. models. Fi leFi eld, which does the work of storing files on file servers like 
AWS CloudFront or S3 for you. Exceptions to this are detailed in Section 6.4.5: When to 
Use BinaryField. 


24.3.2 Getting the Most Out of PostgreSQL 

If using PostgreSQL, be certain that it is set up correctly in production. As this is outside the scope 
of the book, we recommend the following articles: 

>- wiki .postgresql.org/wiki /Detailed_installation_guides 

>- wiki .postgresql.org/wiki /Tuning_Your_PostgreSQL_Server 

>- revsys.com/writings/postgresql-performance.html 

>• craigkerstiens.com/2012/10/01/understanding-postgres-performance 

>- craigkerstiens.com/2013/01/10/more-on-postgres-performance 

For further information, you may want to read the book “PostgreSQL 9.0 High Performance"'. amzn . 
to/lfWctM2 

24.3.3 Getting the Most Out of MySQL 

It’s easy to get MySQL running, but optimizing production installations requires experience and 
understanding. As this is outside the scope of this book, we recommend the following links to help 
you: 
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>■ “The UnojficialMySql Optimizer Guide'’ unofficialmysqlguide.com 
>- “High Performance MySQL” amzn. to/ 188VPcL 


24.4 Cache Queries With Memcached or Redis 

You can get a lot of mileage out of simply setting up Django’s built-in caching system with Mem¬ 
cached or Redis. You will have to install one of these tools, install a package that provides Python 
bindings for them, and configure your project. 

You can easily set up the per-site cache, or you can cache the output of individual views or template 
fragments. You can also use Djangos low-level cache API to cache Python objects. 

Reference material: 

>• docs.dj angoproj ect.com/en/1.11/topics/cache/ 

>- github.com/niwinz/django-redis 


24.5 Identify Specific Places to Cache 

Deciding where to cache is like being first in a long line of impatient customers at Ben and Jerry’s on 
free scoop day. You are under pressure to make a quick decision without being able to see what any 
of the flavors actually look like. 

Here are things to think about: 

>■ Which views/templates contain the most queries? 

>- Which URLs are being requested the most? 

>• When should a cache for a page be invalidated? 

Let’s go over the tools that will help you with these scenarios. 

24.6 Consider Third-Party Caching Packages 

Third-party packages will give you additional features such as: 

>- Caching of QuerySets. 
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>■ Cache invalidation settings/mechanisms. 

>- Different caching backends. 

>• Alternative or experimental approaches to caching. 

A few of the popular Django packages for caching are: 

>- django-cache-machine 
>• johnny-cache 
>- django-cachalot 

See djangopackages.org/grids/g/caching/ for more options. 


WARNING: Third-Party Caching Libraries Aren’t Always the 
Answer 


Having tried many of the third-party Django cache libraries, we have to ask our readers to 
test them very carefully and be prepared to drop them. They are cheap, quick wins, but can 
lead to some hair-raising debugging efforts at the worst possible times. 

Cache invalidation is hard, and in our experience, magical cache libraries are better for 
projects with more static content. By-hand caching is a lot more work, but leads to better 
performance in the long run and doesnt risk those terrifying moments. 


24.7 Compression and Minification of HTML, CSS, and 
JavaScript 

When a browser renders a web page, it usuaUy has to load HTML, CSS, JavaScript, and image files. 
Each of these files consumes the users bandwidth, slowing down page loads. One way to reduce 
bandwidth consumption is via compression and minification. Django even provides tools for you: 
GZi pMiddleware and the {% spaceless %} template tag. Through the at-large Python community, 
we can even use WSGI middleware that performs the same task. 

The problem with making Django and Python do the work is that compression and minification take 
up System resources, which can create bottlenecks of their own. A better approach is to use Apache 
and Nginx web servers configured to compress the outgoing content. If you are maintaining your 
own web servers, this is absolutely the way to go. 
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A common approach is to use a third-party compressiori module or Django library to compress and 
minify the HTML, CSS, and JavaScript in advance. Our preference is django-pipeline which comes 
recommended by Django core developer Jannis Leidel. 

For CSS and JavaScript, many people use JavaScript-powered tools for minification. The advantage 
of this approach is the greater mindshare of tools and solved problems in this domain space. 

Tools and libraries to reference: 

>- Apache and Ngirtx compression modules 

>• django-pipeline 
>- django-compressor 
>■ django-htmlmin 

>- Djangos built-in spaceless tag: docs.djangoproject.com/en/1.11/ref/templates/ 
buiItins/spaceless 

>- djangopackages.org/grids/g/asset-managers/ 


24.8 Use Upstream Caching or a Content Delivery 
NetWork 

Upstream caches such as Varnish are very useful. They run in front of your web server and speed up 
web page or content serving significantly. See varnish-cache.org. 

Content Delivery Networks (CDNs) like Fastly, Akamai, and Amazon Cloudfront serve static me¬ 
dia such as images, video, CSS, and JavaScript files. They usually have servers all over the world, 
which serve out your static content from the nearest location. Using a CDN rather than serving 
static content from your application servers can speed up your projects. 


24.9 Other Resources 

Advanced techniques on scaling, performance, tuning, and optimization are beyond the scope of this 
book, but here are some starting points. 

On general best practices for web performance: 
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>■ YSlow’s Web Performance Best Practices and Rules'. 

developer.yahoo.com/yslow/ 

>■ Google’s Web Performance Best Practices'. 

developers.google.com/speed/docs/best-practices/rules_intro 
On scaling large Django sites: 

>- Written with a focus on scaling Django, the book “High Performance Django'’ espouses many 
good practices. Full of useful information and tricks, as well as questions in each section that 
force you to think about what you are doing. hi ghperformancedj ango. com 
>- David Cramer often writes and speaks about scaling Django at Disqus, Dropbox, and Sentry. 

Readhis blog andkeep aneye out forhis talks, Quora posts, comments, etc. j uste ramer. com 
>- Watch videos and slides from past DjangoCons and PyCons about different develop¬ 
ers’ experiences. Scaling practices vary from year to year and from company to company: 
http://lanyrd.com/search/?q=diango+scaling 


Figure 24.1: With your site running smoothly, you’ll be feeling as cool as a cone. 




TIP: For Sites With High Volume: High Performance Django 


We want to reiterate that “Fligh Performance Django” is worth getting if your site has enough 
traffic to cause issues. While it’s getting old, Peter Baumgartner and Yann Malet wrote the 
book more at the conceptual level, making it a volume that you should considet purchasing. 
>- highperformancedjango.com 

>- amazon.com/High-Performance-Django/dp/1508748128 
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24.10 Summary 

In this chapter we explored a number of bottleneck reduction strategies including: 

>- Whether you should even care about bottlenecks in the first place. 

>- Profiling your pages and queries. 

>- Optimizing queries. 

>- Using your database wisely. 

>- Caching queries. 

>- Identifying what needs to be cached. 

>- Compression of HTML, CSS, and JavaScript. 

>- Exploring other resources. 

In the next chapter, we’ll cover various practices involving asynchronous task queues, which may 
resolve our bottleneck problems. 
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Asynchronous Task Queues 


An asynchronous task queue is one where tasks are executed at a different time from when they 
are created, and possibly not in the same order they were created. Here is an example of a human- 
powered asynchronous task queue: 

O In their spare time, Audrey and Daniel make ice cream cakes, taking orders from friends and 
Family. They use an issue tracker to track their tasks for scooping, spreading, and decorating 
each cake. 

0 Every so often, when they have spare time, they review the list of tasks and pick one to do. 
Audrey prefers scooping and decorating, always doing those tasks first. Daniel prefers scooping 
and spreading, finishing those before decorating. The resuit is asynchronous completion of 
cake-making tasks. 

© As a cake-making task is completed and delivered, they mark the issue as closed. 


TIP: Task Queue vs Asynchronous Task Queue 


In the Django world, both terms are used to describe asynchronous task queue. When some- 
one writes task queue in the context of Django, they usuaUy mean asynchronous task queue. 


Before we get into best practices, let’s go over some definitions: 

Broker The storage for the tasks themselves. This can be implemented using any sort of persistence 
tool, although in Django the most common ones in use are RahhitMQ. and Redis. In the 
human-powered example, the storage is an online issue tracker. 

Producer The code that adds tasks to the queue to be executed later. This is application code, the 
stuff that makes up a Django project. In the human-powered example, this would be Audrey 
and Daniel, plus anyone they can get to pitch in to help. 
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Worker The code that takes tasks from the broker and performs them. Usually there is more than 
one worker. Most commonly each worker runs as a daemon under supervision. In the human- 
powered example, this is Audrey and Daniel. 

Serverless Usually provided by Services such as AWS Lambda, this is, to paraphrase Martin Fowler, 
“where some amount of server-side logic is written by us but unlike traditional architectures 
is run in stateless compute containers that are event-triggered, ephemeral (only last for one 
invocation), and fuUy managed by a 3rd party.” Serverless takes over the role of Broker and 
Worker. In the human-powered example, it’s as if Daniel and Audrey use a third-party Service 
to take the orders and then follow their precise instructions on doing the work. 

25.1 Do We Need a Task Queue? 

It depends. They add complexity but can improve user experience. Arguably it comes down to whether 

a particular piece of code causes a bottleneck and can be delayed for later when more free CPU cycles 

are available. 

Here is a useful rule of thumb for determining if a task queue should be used: 

Results take time to process: Task queue should probably be used. 

Users can and shotdd see results immediately: Task queue should not be used. 


Let’s go over some possible use cases: 


1 Issue 

Use Task Queue? 1 

Sending bulk email 

Yes 

Modifying files (including images) 

Yes 

Fetching large amounts of data from third-party Ice Cream APIs 

Yes 

Inserting or updating a lot of records into a table 

Yes 

Updating a user profile 

No 

Adding a blog or CMS entry 

No 

Performing time-intensive calculations 

Yes 

Sending or receiving of webhooks 

Yes 

Table 25.1: Should a Project Have a Task Queue? 


336 





25.2: Choosing Task Queue Software 


Please keep in mind there are site-trafRc driven exceptions to all of these use cases: 

>- Sites with small-to-medium amounts of trafRc may never need a task queue for any of these 
actions. 

>■ Sites with larger amounts of traflic may discover that nearly every user action requires use of a 
task queue. 

Determining whether or not a site or action needs a task queue is a hit of an art. There is no easy 
answer we can provide. However, knowing how to use them is a really powerful tool in any developer’s 
toolkit. 


25.2 Choosing Task Queue Software 

Celery, Django Channels, or calls to serverless Services such as AWS Lambda, which to choose? 
Let’s go over their pros and cons: 


Software Pros Cons 


Celery 

A Django and Python Standard, 
many different storage types, 
flexihle, full-featured, great for 
high volume 

ChaUenging setup, steep 
learning curve for anything 
hut the hasic stuff 

Dj angoChannels 

Defacto Django Standard, 
flexihle, easy-to-use, adds 
wehsocket support to Django 

No retry mechanism, 
Redis-only 

AWSLambda 

Flexihle, scalahle, easy setup 

API call can he slow, 
requires external logging 
Services, adds complexity, 
requires creating REST 

API for notifications 

Redis-Queue, Huey, other 
Django-friendly queues 

Lower memory footprint than 
Celery, relatively easy setup 

Not as many features as 
Celery, usually Redis-only, 
smaUer communities 
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Software Pros Cons 


django-background-tasksVery easy Setup, easy to use, Uses Django ORM for 

Works on Windows, good for backend, absolutely terrible 

small volume or batch jobs, uses for medium-to-high 
Django ORM for backend volume 

Table 25.2: Comparing Task Queue Software 


Here is our general rule of thumb: 

>- If time permits, move ali asynchronous processes to Serverless systems such as AWS Lambda. 

>• If API calls to Serverless become an issue, encapsulate these calls in Celery tasks. For us, this 
has only been a problem with bulk API calls to AWS Lambda. 

>■ Use Django Channels for websockets. The lack of retry mechanism forces you to invent things 
that Celery provides out-of-the-box. 

>- For security and performance reasons, any and ali API calls to user-defined URLs are done 
through task queues. 


Of course, your own experience and knowledge should be used to determine which task queue system 
you use for a project. Examples: 


>- If you have a good amount of Celery experience and are comfortable with it, then by all means 
use it for smaU volume or toy projects. 

>■ Most Serverless systems have hard-limits on disk drive space (Example: AWS Lambda limits 
you to 512MB). This can be a problem when manipulating large files (transcoding of video) 
or using certain libraries. In these cases, you can either use third-party Services or construet 
dedicated servers running Celery to handle such tasks. 

>- The extensibility of the Django Channels Generic Consumers are so nice that weVe been 
tempted to write our own retry mechanisms. While we havent had the time to do it, you 
might. Just be aware that it’s a larger, more complicated task than you might expect. 

>• In theory, queues like Redis Queue and Huey have easier setup than Celery. However, project 
templates like Cookiecutter Django render this advantage moot. 
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25.3 Best Practices for Task Queues 

While each of the different task queue packages has their own quirks, there are some constants we 
can apply to all of them. A nice feature about these practices is that they help with the portability 
of your task functions. This can be incredibly useful when you discover that while Django Channels 
has been useful, the lack of a retry mechanism requires you to switch to Celery. 


25.3.1 Treat Tasks Like Views 

Throughout this book we recommend that views contain as little code as possible, calling methods 
and functions from other places in the code base. We believe the same thing applies to tasks. 

A common trap is for the code inside task functions to become long and ugly, because the assumption 
is that “the task queue hides it from the user.” WeVe been guilty of this ourselves. To avoid this, you 
can put your task code into a function, put that function into a helper module, and then call that 
function from a task function. 

All task queue packages do some kind of serialization/abstraction of our task functions and their 
arguments. This makes debugging them much more difficult. By using our task functions to call 
more easily tested normal functions, we not only make writing and debugging our code easier, we 
also encourage more reuse. 

The same goes for Serverless code. Rather than put a lot of logic into our AWS lambda functions, 
we create installable, testable packages that we import from. This means a gigantic reduction of 
production-style debugging. 


25.3.2 Tasks Aren’t Free 

Remember that the memory and resources to process a task have to come from somewhere. Overly 
resource-heavy tasks might be hidden, but they can stili cause site problems. 

Even if resource-intensive code is executed from a task, it should stili be written as cleanly as possible, 
minimizing any unnecessary resource usage. Optimization and profiling can help here. 
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Even Serverless tasks are not free. Remember, the term ‘Serverless’ is a misnomer, the code is being 
run in servers. Slow Serverless-tasks can literally run out of time or create a surprisingly large biU at 
the end of the month. 


25.3.3 Only Pass JSON-Serializable Values to Task Functions 

Just like views, for task function arguments, only pass JSON-serializable values. That limits us to 
integers, floats, strings, lists, tuples, and dictionaries. Don’t pass in complex objects. Here’s why: 

O Passing in an object representing persistent data. For example, ORM instances can cause a race 
condition. This is when the underlying persistent data changes before the task is run. Instead, 
pass in a primary key or some other identifier that can be used to call fresh data. 

0 Passing in complex objects that have to be serialized into the task queue is time and memory 
consuming. This is counter-productive to the benefits we’re trying to achieve by using a task 
queue. 

© WeVe found debugging JSON-serializable values easier than debugging more complex objects. 

O Depending on the task queue in use, only JSON-serializable primitives are accepted. 

25.3.4 Write Tasks as Idempotent Whenever Possible 

When we say idempotent (en. wi ki pedi a . org/wi ki /Idempotence) we mean that you can run 
the task multiple times and get the same resuit. This is important with task queues because retries 
are expected, even with successfully completed tasks (not uncommon with broker restarts). When a 
retry, intentional or not, occurs, you want the task to respond with the same resuit each time it runs. 


TIP: Pure Functions Over Idempotent Functions 


Nathan Cox, Djangonaut and bleeding edge language enthusiast, encourages us to write tasks 
using pure functions (en . wi ki pedi a .org/wi ki /Pure_f uncti on). The main difference 
being: 

>- pure functions either do not aUow or strongly discourage side effects, while 
>- idempotent functions dont mind if there are side effects just so long as the direct resuit 
is the same over two function calls. 

This may seem like a fine distinction, but it’s worth keeping pure functions in mind when 
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writing idempotent tasks as it encourages us to write more straight-forward task code. Con- 
sidering the complexity that task queues can bring to a project, we should embrace anything 
we can do to make ourselves write cleaner asynchronous code. 


25.3.5 Don’t Keep Important Data in Your Queue 

Except for Django Channels, ali the asychronous task queue options weVe presented include a built- 
in retry mechanism. This is great, but sometimes even the retries fail. This can occur for any reason, 
most commonly bugs within our own code or encountering latency when communicating with third- 
party APIs. What this means is that critical tasks can fail to run. WeVe seen this occur with billing 
customers, sending emails, or making reservations. 

The solution is to track the status of an action within the alfected record(s). For example, as a customer 
is about to be billed, mark them as not having been biUed yet, then call the task. If the task succeeds, 
have it update the customer has having been billed. If the task fails, then it will fail to update the 
customer and a simple query will reveal the customer hasnt yet paid their bili. 

If you want to know more, Dan Poirier of Caktus wrote an excellent article about this technique: 

>• http://blt.ly/2eqdlDZ 

25.3.6 Learn How to Monitor Tasks and Workers 

Gaining visibili ty into the status of tasks and workers is critical for debugging of task functions. Some 
useful tools: 

>- Celery: pypl. python. org/pypl /flower 

25.3.7 Logging! 

Since task queues are working “behind the scenes,” it can be hard to determine exactly what is going 
on. This is where logging (Chapter 27: Logging: Whats It For, Anyway?) and tools like Sentry 
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become really useful. In error-prone task code, it can be a good idea to log inside of each task function. 
This will make debugging production code easier. 

When using Serverless tasks, weVe found that Sentry isn’t an option, it is an absolute necessity. When 
you hit that obscure edge case no amount of logging will capture the depth of data that Sentry 
provides. 


25.3.8 Monitor the Backlog 

As traffic increases, tasks can pile up if there arent enough workers. When we see this happening, 
it’s time to increase the number of workers. Of course, this doesnt apply to Serverless tasks. 


25.3.9 Periodically Ciear Out Dead Tasks 

Sometimes a task is passed into a queue and then just sits there doing nothing for some reason. It 
could be caused by a bug, e.g. a resource being used by the task might no longer exist. However these 
things happen, they can build up over time, taking up space in our system. 

Learn how your Software cleans out dead tasks, and check to make sure it’s running properly. Of 
course, this doesnt apply to Serverless tasks. 

25.3.10 Ignore Results We Don’t Need 

When a task completes, the broker is designed to record whether it succeeded or failed. While use¬ 
ful for statistical purposes, this exit status is not the resuit of the job the task was performing. As 
recording this status takes up time and storage space, it’s a feature we usually turn off. 


25.3.11 Use the Queue’s Error Handling 

What happens when a task fails? It can be caused by a network error, a third-party API going down, 
or anything else that can be imagined. Look up how to do the following for your task queue Software 
and learn how to set them: 
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>■ Max retries for a task 
>- Retrydelays 

Retry delays deserve a lot of consideratiori. When a task fails, we like to wait at least 10 seconds 
before trying again. Even better, if the task queue Software allows it, increase the delay each time an 
attempt is made. We set things this way in order to give the conditions that caused a failure to resolve 
themselves. 


25.3.12 Leam the Features ofYour Task Queue Software 

Celery, Django Channels, and Redis Queue allow for definition of multiple queues. In fact, Celery 
has fancy routing features that no other Software package possesses. 

If we don’t take the time to explore, learn, and use these features, we’re losing out on lots of secret 
sauce. Staying ignorant of these features can mean that instead of leaning on our package of choice, 
we end up writing code that duplicates what the package provides. 

In fact, while weVe become fans of using Boto3 to call AWS Lambda to perform tasks, half the 
reason we don’t let go of Celery is because it gives us so much control over execution. 

25.4 Resources for Task Queues 


General: 

► VInta. com.br/blog/2016/database-concurrency-in-dj ango-the-right-way/ 
Essential reading! 

► fullstackpython.com/task-queues.html 

► slideshare.net/bryanhelmig/task-queues-comorichweb-12962619 

► github.com/carijm/dj ango-transaction-hooks 

Django database backends that permit registering post-transaction-commit hooks. 

Celery: 

► celeryproj ect. com Homepage of Celery 

► denibertovic.com/posts/celery-best-practices/ Must-read article for anyone 
learning Celery 
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>■ pypi . python. org/pypi /flower A web-based tool for managing Celery clusters 
>- wiredcraft.com/blog/3-gotchas-for-celery 

>• caktusgroup. com/blog/tags/celery / The Caktus blog has a number of incredibly use- 
ful articles on Celery. 

Django Channels 

>- channels. readthedocs. io Homepage of Django Channels 
>• github. com/dj ango/channels Source repo 


25.5 Summary 

In this chapter we explored high-level practices for working with task queues. Because of the ab- 
straction involved in using them, we advocate treating them like views, minimizing the amount of 
business logic within. 

We also covered the use of Serverless tasks through mostly the lens of AWS Lambda. It’s an exciting 
new way of doing things, but the limitations can be overwhehning. 

In the next chapter, we’ll go over the basies of securing Django projects. 


344 




26 


Security Best Practices 


When it comes to security, Django has a pretty good record. This is due to security tools provided 
by Django, solid documentation on the subject of security, and a thoughtful team of core developers 
who are extremely responsive to security issues. However, it’s up to individual Django developers 
such as ourselves to understand how to properly secure Django-powered applications. 

This chapter contains a Hst of things helpful for securing your Django application. This list is by no 
means complete. Consider it a starting point. 


TIP: What to Do ifYou Have a Security Breach 


If youre in the midst of a security crisis, please go to Appendix H: Handling Security Failures. 


26.1 Reference Security Sections in Other Chapters 

A number of other chapters in this book contain dedicated security sections, or touch on security 
matters. These are found at the following locations: 

>- Section 5.3: Separate Configuration From Code 

>- Section 12.3: Always Use CSRF Protection With HTTP Forms That Modify Data 
>- Section 17.5.1: Set settings.CSRF_COOKIE_HTTPONLY Appropriately 
>- Section 26.27: Never Display Sequential Primary Keys 
>■ Section 19.8: Secure the Django Admin 
>■ Appendix H: Handling Security Failures 
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26.2 Harden Your S ervers 

Search online for instructions and checklists for server hardening. Server hardening measures in¬ 
clude but are not limited to things like setting up firewaUs (help. ubuntu . com/communi ty/UFW), 
changing your SSH port, and disabling/removing unnecessary Services. 

26.3 KnowDjango’s Security Features 

Django 1.11’s security features include: 

>• Cross-site scripting (XSS) protection. 

>• Cross-site request forgery (CSRF) protection. 

>■ SQL injection protection. 

>- Clickjacking protection. 

>- Support for TLS/HTTPS/HSTS, including secure cookies. 

>- Secure password storage, using the PBKDF2 algorithm with a SHA256 hash by default. 

>- Automatic HTML escaping. 

>■ An expat parser hardened against XML bomb attacks. 

>- Hardened JSON, YAML, and XML serialization/deserialization tools. 

Most of Djangos security features “just work” out of the box without additional configuration, but 
there are certain things that you’ll need to configure. WeVe highlighted some of these details in this 
chapter, but please make sure that you read the oflicial Django documentation on security as well: 
docs.dj angoproj ect.com/en/1.11/topics/security/ 

26.4 Turn OfFDEBUG Mode in Production 

Your production site should not be running in DEBUG mode. Attackers can find out more than 
they need to know about your production setup from a helpful DEBUG mode stack trace page. For 
more information, see docs. dj angoproj ect. com/en/1.11/ ref/setti ngs/#debug. 

Keep in mind that when you turn olf DEBUG mode, you will need to set ALLOWED_HOSTS 
or risk raising a SuspiciousOperation error, which generates a 500 error that can be hard to debug. 
For more information on setting/debugging ALLOWED_HOSTS see: 

>- Section 26.7: Use Allowed Hosts Validation 

>- Section 33.2.4: That Troublesome setti ngs .ALL0WED_H0STS Error 
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26.5 Keep Your Secret Keys Secret 

If your SECRET_KEY setting is not secret, this means you risk everything from remote code 

exeeution to password hacking. Your API keys and other secrets should be carefuUy guarded as weU. 
These keys should not even be kept in version control. 

We cover the mechanies of how to keep your SECRET_KEY out of version control in Chapter 5: 

Settings and Requirements Files, Section 5.3: Separate Configuration From Code, and Section 5.4: 
When You Can’t Use Environment Variables. 


26.6 HTTPS Everywhere 

It is always better to deploy a site behind FITTPS. Not having FITTPS means that malicious network 
users can sniff authentication credentials between your site and end users. In fact, all data sent between 
your site and end users is up for grabs. 

There is also no guarantee that any of your users are seeing what you expect them to see: an attacker 
could manipulate anything in the request or the response. So FITTPS makes sense even if all your 
Information is public, but you do care about the integrity of the information. 

Your entire site should be behind FITTPS. Your site’s static resources should also be served via 
HTTPS, otherwise visitors will get warnings about “insecure resources” which should rightly scare 
them away from your site. For reference, these warnings exist because they are a potential man-in- 
the-middle vector. 


TIP: Jacob Kaplan-Moss on HTTPS vs HTTP 


Django coTeader Jacob Kaplan-Moss says, “Your whole site should only be available via 
HTTPS, not HTTP at aU. This prevents getting “firesheeped” (having a session cookie stolen 
when served over HTTP). The cost is usually minimal.” 


If visitors try to access your site via HTTP, they should be redirected to HTTPS. This can be done 
either through configuration of your web server or with Django middleware. Performance-wise, it’s 
better to do this at the web server level, but if you don’t have control over your web server settings 
for this, then redirecting via Django middleware is fine. 
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You should obtain an SSL certificate from a reputable source rather than creating a self-signed cer- 
tificate. To set it up, follow the instructions for your particular web server or platform-as-a-service. 
Our preferred Service is the very reputable (and free) letsencrypt. org. This Service makes it so 
easy to create SSL certificates there is no reason to use self-signed certificates. 


TIP: Please Use Let’s Encrypt for SSL Certificates 


It used to be that getting an SSL certificate was a painful process. The tools and documenta- 
tion were and are unpleasant to use, and some of the companies provided them covered their 
sites with so many shady advertisements that you wondered if they were run by criminals. 
That aU changed in April of 2016 with the launch of Let’s Encrypt (letsencrypt.org). 
It’s a free Service sponsored by large and small organizations who had the same problems we 
did. They provide easy-to-use open source Software that is very well documented and used 
for millions of projects. 

Going forward, our opinion is that unless you are using a trusted Service that integrates with 
your hosting platform, you should be using Let’s Encrypt. An example of this would be 
Amazons Certificate Manager for EC2-based projects. 


TIP: Use django.middleware.security.SecurityMiddleware 


The tool of choice for projects on Django 1.11+ for enforcing HTTPS/SSL across an entire 
site through middleware is built right in. To activate this middleware just follow these steps: 
O Add django. middleware. security. Security Middleware to the 

settings.MIDDLEWARE_CLASSES definition. 

0 Set settings.SECURE_SSL_HOST to True . 


WARNING: django.middleware.security.SecurityMiddleware Does Not 
Include static/media 


Even if aU Django requests are served over HTTPS, omitting HTTPS for resources like 
javascript would stiU allow attackers to compromise your site. 

As JavaScript, CSS, images, and other static assets are typicaUy served directly by the web 
server (nginx, Apache), make certain that serving of such content is done via HTTPS. 
Providers of static assets such as Amazon S3 now do this by default. 
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26.6: HTTPS Everywhere 


26.6.1 Use Secure Cookies 


Your site should inform the target browser to never send cookies unless via HTTPS. You’ll need to 
set the foUowing in your settings: 



Read docs. dj angoproj ect. com/en/1.11/topi cs/securi ty/#ssl-https for more details. 


26.6.2 Use HTTP Striet Transport Security (HSTS) 

HSTS can be configured at the web server level. Follow the instructions for your web server, platform- 
as-a-service, and Django itself (via settings.SECURE_HSTS_SECONDS ). 

If you have set up your own web servers, Wikipedia has sample HSTS configuration snippets that 
you can use: en.wikipedi a.org/wiki/HTTP_Strict_Transport_Security 

When you enable HSTS, your site’s web pages include a HTTP header that teUs HSTS-compliant 
browsers to only connect to the site via secure connections: 

>• HSTS-compliant browsers wiU redirect HTTP links to HTTPS. 

>- If a secure connection isn’t possible (e.g. the certificate is self-signed or expired), an error mes- 
sage wiU be shown and access will be disaUowed. 

To give you a better idea of how this works, here’s an example of what a HTTP Striet Transport 
Security response header might look like: 



Some HSTS configuration advice: 
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O You should use HSTS’ includeSubDomains mode if you can. This prevents attacks involving 
using non-secured subdomains to write cookies for the parent domain. 

0 Set max-age to a small value like 3600 (1 hour) during initial deployment of a secured site to 
make sure you havent screwed something up or forgotten to make some portion of the site 
available via HTTPS. We suggest this small value because once you set max-age, you eant 
unset it for users; their browsers control expiration, not you. 

© Once you ve confirmed that your site is properly secured, set max-age to a large value like 
31536000 (12 months) or 63072000 (24 months) if you can. 


WARNING: Choose Your HSTS Policy Duration Carefiilly 


Remember that HSTS is a one-way switch. It’s a declaration that for the next N seconds, 
your site wiU be HTTPS-only. Dont set a HSTS policy with a max-age longer than you are 
able to maintain. Browsers do not offer an easy way to unset it. 


Note that HSTS should be enabled in addition to redirecting all pages to HTTPS as described earlier. 


WARNING: Additional Warning for includeSubDomains 


We recommend everyone to use HSTS with a long duration and to use includeSubDo¬ 
mains. However, especially in projects with lots of legacy components, the combination 
requires great care when configuring. 

Example: Imagine we create a new Django website called example.com. Of course, the site 
is HTTPS with HSTS. We test the HSTS settings, which work fine, and then increase the 
duration to a year. Alas, after a month, someone realises legacy. example. com is stili a production 
Service and does not support HTTPS. We remove includeSubdomains from the header, 
but by now it’s aheady too late: all clients inside the company have the old HSTS header 
remembered. 

In short, before even considering includeSubdomains, one should be entirely aware of what 
might be hosted under the domain that HSTS is configured on. 
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26.6.3 HTTPS Configuration Tools 

Mozilla provides a SSL configuration generator at the mozilla.github.io/server-side-t Is/ 
ssl-conf 1 g- generator/, which can provide a starting point for your own configuration. While 
not perfect, it expedites setting up HTTPS. As our security reviewers say, “In general, any HTTPS 
is better than plain HTTP” 

Once you have a server set up (preferably a test server), use the Qualys SSL Labs server test at 
sllabs.com/ssltest/ to see how well you did. A fun security game is trying to score an A+. 
Especially as the official Two Scoops of Django reward for getting that good of a grade is a trip to 
the local favorite ice cream saloon. 


26.7 Use Allowed Hosts Validation 

In production, you must set ALLOWEDHOSTS in your settings to a list of allowed host/domain 
names in order to avoid raising SuspiciousOperation exceptions. This is a security measure to 
prevent the use of fake HTTP host headers to submit requests. 

We recommend that you avoid setting wildcard values here. Eor more information, read the Django 
documentation on ALLOWED_HOSTS and the get_host() method: 

>• docs.dj angoproj ect.com/en/1.11/ref/settings/#allowed-hosts 
>- docs.dj angoproj ect.com/en/1.11/ref/request-response/#dj ango.http. 
HttpRequest.get_host 


26.8 Always Use CSRF Protection With HTTP Forms That 
Modify Data 

Django comes with easy-to-use cross-site request forgery protection (CSRE) built in, and by default 
it is activated across the site via the use of middleware. We have some strong recommendations 
discussed in Section 12.3: Always Use CSRE Protection With HTTP Eorms That Modify Data. 

26.9 Prevent Against Cross-Site Scripting (XSS) Attacks 

XSS attacks usuaUy occur when users enter malignant JavaScript that is then rendered into a template 
directly. This isn’t the only method, but it is the most common. Eortunately for us, Django by default 
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escapes <, and &, which is all that is needed for proper HTML escaping. 
The following are recommended by the Django security team: 


26.9.1 Use format_html Over mark_safe 

Django gives developers the ability to mark content strings as safe, meaning that Django’s own 

safeguards are taken away. A better alternative is django. utils.html. format_html, which is like 

Pythons str. format () method, except designed for building up HTML fragments. All args and 
kwargs are escaped before being passed to str. format () which then combines the elements. 

Reference: docs.dj angoproj ect.com/en/1.ll/ref/utils/#dj ango.uti Is.html. 

format_html 

26.9.2 Don’t AllowUsers to Set Individual HTML Tag Attributes 

If you aUow users to set individual attributes of HTML tags, that gives them a venue for injecting 
malignant JavaScript. 


26.9.3 Use JSON Encoding for Data Consumed byJavaScript 


Rely on JSON encoding rather than finding ways to dump Python structures directly to templates. 
It’s not just easier to integrate into client-side JavaScript, it’s safer. 


26.9.4 Beware Unusual JavaScript 

Due to JavaScripts weird semantics, it’s possible to construet syntactically-valid, 
executable programs from a very tiny subset of characters. Per jazcash.com/ 
a-javascript-journey-with-only-six-characters, it’s possible to transform regular- 
looking JavaScript into an alphabet of only sbc characters (plus sign, exclamation mark, open/close 
bracket and open/close parenthesis). 
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26.9.5 Add Content Security Policy Headers 

Also known as CSP, Content Security Policy provides a Standard method to declare approved origins 
of content that browsers should be allowed to load on a website. Covered types are JavaScript, CSS, 
HTML frames, web workers, fonts, images, embeddable objects such as Java applets, ActiveX, audio 
and video files, and other HTML5 features. Even with django-csp, implementing this isn’t trivial as 
it requires standing up a reporting Service or using a third-party Service. 

>• en.wikipedia.org/wiki /Content_Security_Policy 
>- github.com/mozilla/django-csp 

26.9.6 Additional Reading 

There are other avenues of attack that can occur, so educating yourself is important. 

>• docs.dj angoproj ect.com/en/1.11/ref/templates/buiItins/#escape 
>- en.wikipedia.org/wiki /Cross-site_scripting 

26.10 Defend Against Python Code Injection Attacks 

We once were hired to help with a project that had some security issues. The requests coming into the 
site were being converted from django.http.HttpRequest objects directly into strings via Creative 
use of the str() function, then saved to a database table. Periodically, these archived Django requests 
would be taken from the database and converted into Python dicts via the eval() function. This 
meant that arbitrary Python code could be run on the site at any time. 

Needless to say, upon discovery the critica! security flaw was quickly removed. This just goes to show 
that no matter how secure Python and Django might be, we always need to be aware that certain 
practices are incredibly dangerous. 


26.10.1 Python Built-Ins That Execute Code 

Beware of the eval() , exec() , and execfile() built-ins. If your project allows arbitrary strings or 
files to be passed into any of these functions, you are leaving your system open to attack. 
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For more information, read “Eval Really Is Dangerous” by Ned Batchelder: 
nedbatchelder.com/blog/201206/eval_really_is_dangerous.html 


26.10.2 Python Standard Library Modules That Can Execute Code 


“Never unpickle data receivedfrom an untrusted or unauthenticatedsource.” 

- docs.python.org/3/library/pickle.html 

You should not use the Python Standard library’s pi ckle module to deserialize anything which could 
have been modified by the user. As a general rule, avoid accepting pickled values from user for any 
reason. Specific warnings about pi ckle and security are listed below: 

>• lincolnloop.com/blog/playing-pickle-security/ 

>• blog.nelhage.com/2011/03/exploiting-pickle/ 

26.10.3 Third-Party Libraries That Can Execute Code 

When using PyYAML, only use safe_load() . While the use of YAML in the Python and Django 

communities is rare outside of continuous integration, it’s not uncommon to receive this format 
Therefore, if you are accepting YAML documents, only load them with the 
method. 

For reference, the yamLload() method will let you create Python objects, which is really bad. As Ned 

Batchelder says, yamLload() should be renamed to yaml.dangerous_load() : nedbatchelder. 

com/blog/201302/war_is_peace.html 


from other Services, 
yaml. safe_load () 


26.10.4 Be Careful With Cookie-Based Sessions 

Typically most Django sites use either database- or cache-based sessions. These function by storing 
a hashed random value in a cookie which is used as a key to the real session value, which is stored 
in the database or cache. The advantage of this is that only the key to the session data is sent to the 
client, making it very chaUenging for malignant coders to penetrate Django’s session mechanism. 
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However, Django sites can also be built using cookie-based sessions, which place the session data 
entirely on the client’s machine. While this means slightly less storage needs for the server, it comes 
with security issues that justify caution. SpecificaUy: 

O It is possible for users to read the contents of cookie-based sessions. 

0 If an attacker gains access to a project’s SECRET_KEY and your session serializer is JSON- 

based, they gain the ability to falsify session data and therefore, if authentication is used, im- 
personate any user. 

© If an attacker gains access to a projects SECRET_KEY and your session serializer is pickle- 

based, they gain the ability to not only falsify session data and also execute arbitrary code. In 
other words, not only can they assume new rights and privileges, they can also upload working 
Python code. If you are using pickle-based sessions or are considering using them, please read 
the tip below. 

O Another disadvantage of this configuration is that sessions can’t be invalidated in a guaranteed 
way (except when they expire): you can try to override the cookie in the browser with a new 
value, but you can’t enforce an attacker to use it: if they continue sending requests with the old 
cookie, the session backend wont know the difference. 


TIP: Use JSON for Cookie-Based Sessions 


The default cookie serializer for Django is JSON-based, meaning that even if an attacker 
discovers a projects SECRET_KEY , they eant execute arbitrary code. If you decide to 
write your own cookie serializer, stick to using JSON as the format. 

Never, ever use the optional pickle serializer. 

Resources on the subject: 

>- docs.djangoproject.com/en/1.11/topics/http/sessions/ 

#session-seri alization 

>- docs.dj angoproj ect.com/en/1.11/ref/settings/#session-serializer 


Another thing to consider is that cookie-based sessions are a potential client-side performance bot- 
tleneck. Transmitting the session data server-to-client is generaUy not an issue, but client-to-server 
transmissions are much, much slower. This is literally the difference between download and upload 
speeds aU internet users encounter. 

In general, we try to avoid cookie-based sessions. 
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Additional reading: 

>• docs.dj angoproj ect.com/en/1.11/topics/http/sessions/ 

#using-cookie-based-sessions 
>■ http://bit. ly/2plfHqU Threatpost.com article on cookies 
>- yuiblog.com/blog/2007/03/01/performance-research-part-3/ 

26.11 Validate All Incoming Data With Django Forms 

Django forms should be used to validate all data being brought into your project, including from 
non-web sources. Doing so protects the integrity of our data and is part of securing your application. 
We cover this in Section 12.1: Validate AU Incoming Data With Django Forms. 


TIP: Using DRF Serializers Instead of Django Forms 


Django REST Framework’s validation is as well constructed and secure as Django’s form 
libraries. If you are more familiar with DRF, then using serializers to validate all incoming 
data is perfectly okay. 


26.12 Disable the Autocomplete on Payment Fields 

You should disable the FITML field autocomplete browser feature on fields that are gateways to 
payment. This includes credit card numbers, CWs, PINs, credit card dates, etc. The reason is that a 
lot of people use public computers or their personal computers in public venues. 

For reference, Django forms make this easy: 



For any site that might be used in a public area (an airport for example), consider cbanging the form 
field itself to Passwordinput: 
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26.13 Handle User-Uploaded Files Carefully 


The only way to completely safely serve user-provided content is from a completely separate domain. 
For better or worse, there are an infinite number of ways to bypass file type validators. This is why 
security experts recommend the use of content delivery networks (CDNs): they serve as a place to 
store potentially dangerous files. 


If you must allow upload and download of arbitrary file types, make sure that the server uses the 
“Content-Di sposi ti on: attachment” header so that browsers won’t display the content inline. 


26.13.1 When a CDN Is Not an Option 


When this occurs, uploaded files must be saved to a directory that does not allow them to be executed. 
In addition, at the very least make sure the HTTP server is configured to serve images with image 
content type headers, and that uploads are restricted to a whitelisted subset of file extensions. 


Take extra care with your web servers configuration here, because a malicious user can try to attack 
your site by uploading an executable file like a CGI or PHP script and then accessing the URL. This 
wont solve every problem, but it’s better than the defaults. 


Consuit your web server’s documentation for instructions on how to configure this, or consuit the 
documentation for your platform-as-a-service for details about how static assets and user-uploaded 
files should be stored. 
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26.13.2 Django and User-Uploaded Files 

Django has two model fields that allow for user uploads: Fi leFi eld and ImageFi eld. They come 
with some built-in validation, but the Django docs also strongly advise you to “pay close attention to 
where youre uploading them and what type of files they are, to avoid security holes.” 

If you are only accepting uploads of certain file types, do whatever you can do to ensure that the user 
is only uploading files of those types. For example, you can: 

>- Use the python-magic library to check the uploaded file’s headers: 
github.com/ahupp/python-magic 

>• Validate the file with a Python library that specifically works with that file type. Unfortunately 
this isn’t documented, but if you dig through Django’s ImageFi eld source code, you can see 
how Django uses PIL to validate that uploaded image files are in fact images. 

>- Use defusedxml instead of native Python XML libraries or kml. See Section 26.21: Guard 
Against XML Bombing With defusedxml. 


WARNING: Custom Validators Aren’t the Answer Here 


Doht just write a custom validator and expect it to validate your uploaded files before dan- 
gerous things happen. Custom validators are run against field content after theyve already 
been coerced to Python by the fields to_python () method. 

If the contents of an uploaded file are malicious, any validation happening after 
to_python () is executed may be too late. 


Further reading: 

>• docs.dj angoproj ect.com/en/1.11/ref/models/fields/#filefield 

26.14 Don’t Use ModelForms.Meta.exclude 

When using ModelForms, always use Meta, fields. Never use Meta. exclude. The use of 
Meta. exclude is considered a grave security risk, specifically a Mass Assignment VulnerabiUty. 
We eant stress this strongly enough. Dont do it. 
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One common reason we want to avoid the Meta . exclude attribute is that its behavior implicitly 
allows ali model fields to be changed except for those that we specify. When using the excludes 
attribute, if the model changes after the form is written, we have to remember to change the form. 
If we forget to change the form to match the model changes, we risk catastrophe. 

Let’s use an example to show how this mistake could be made. We’ll start with a simple ice cream 
store model: 



Here is the wrongway to define the Model Form fields for this model: 


Example 26.6: Implicit Definition of Form Fields 

# DON'T DO THIS! 

from django import forms 

from .models import Store 

class StoreForm(forms.ModelForm): 

class Meta: 

model = Store 

# DON'T DO THIS: Implicit definition of fields. 

# Too easy to make mistakes! 

excludes = ("pk", "slug", "modified", "created", "owner") 


In contrast, this is the right way to define the same ModelForms fields: 
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The first code example, as it involves less typing, appears to be the better choice. It’s not, as when 
you add a new model field you now you need to track the field in multiple locations (one model and 
one or more forms). 

Let’s demonstrate this in action. Perhaps after launch we decide we need to have a way of tracking 
store co-owners, who have aU the same rights as the owner. They can access account information, 
change passwords, place orders, and specify banking information. The store model receives a new 
field as shown on the next page: 
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26.15: Dont Use ModelForms.Meta.fields = ”_all_” 

The first form code example which we warned against using relies on us remembering to alter it to 
include the new co_owners field. Ifwe forget, then anyone accessing that store’s HTML form can 
add or remove co-owners. While we might remember a single form, what if we have more than one 
HodelForm for a model? In complex applications this is not uncommon. 

On the other hand, in the second example, where we used Meta. fi elds we know exactly what 
fields each form is designed to handle. Changing the model doesnt alter what the form exposes, and 
we can sleep soundly knowing that our ice cream store data is more secure. 


26.14,1 Mass Assignment Vulnerabilities 

The problem we describe in this section is a Mass Assignment Vulnerability. 

These occur when the patterns such as Active Record, designed to empower developers, create se- 
curity risks for web applications. The solution is the approach we advocate in this section, which is 
explicit definition of fields that can be modified. 

See n. wi ki pedi a. org/wi ki /Mass_assi gnment_vulnerabi li ty for more detail. 

26.15 Don’tUse ModelForms.Meta.fields = ”_all_” 

This includes every model field in your model form. It’s a shortcut, and a dangerous one. It’s very 
similar to what we describe in Section 26.14: Dont Use ModelForms.Meta.exclude, and even with 
custom validation code, exposes projects to form-based Mass Assignment Vulnerabilities. We advo¬ 
cate avoiding this technique as much as possible, as we feel that it’s simply impossible to catch all 
variations of input. 

26.16 Beware of SQL Injection Attacks 

The Django ORM generates properly-escaped SQL which will protect your site from users attempt- 
ing to execute malignant, arbitrary SQL code. 

Django allows you to bypass its QRM and access the database more directly through raw SQL. When 
using this feature, be especiaUy careful to escape your SQL code properly. This is of concern in these 
specific components of Django: 
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>■ The .raw() ORM method. 

>- The . extra 0 ORM method. 

>• Directly accessing the database cursor. 

Reference: 

>• docs.dj angoproj ect.com/en/1.11/topics/security/ 

#sql-i nj ection-protectiori 

26.17 Never Store Credit Card Data 

Unless you have a strong understanding of the PCI-DSS security standards 
(pcisecuritystandards.org) and adequate time/resources/funds to validate your PCI 
compliance, storing credit card data is too much of a liability and should be avoided. 

Instead, we recommend using third-party Services like Stripe, Braintree, Adyen, PayPal, and others 
that handle storing this information for you, and allow you to reference the data via special tokens. 
Most of these Services have great tutorials, are very Python and Django friendly, and are well worth 
the time and effort to incorporate into your project. 


TIP: Educate Yourself on PCI Compliance 


Ken Cochrane has written an excellent blog post on PCI 
compliance. Please read kencochrane.net/blog/2012/01/ 

developers-guide-to-pci-compliant-web-applications/ 


TIP: Read the Source Code of Open Source E-Commerce 
Solutions 


If you are planning to use any of the existing open source Django e-commerce Solutions, 
examine how the solution handles payments. If credit card data is being stored in the database, 
even encrypted, then please use another solution. 


26.18 Monitor Your Sites 

Check your web servers’ access and error logs regularly. Install monitoring tools and check on them 
frequently. Keep an eye out for suspicious activity. 
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26.19 Keep Your Dependencies Up-to-Date 

You should always update your projects to workwith the latest stable release of Django and third- 
party dependencies. This is particularly important when a release includes security flxes. For that, we 
recommend pyup . i o, which automaticaUy checks requirements files against the latest versions that 
PyPI provides. 

‘IVe set up (these kinds of Services) for distinet actions: it mails me once a week for 
each project with any outdated dependencies, and if it finds an insecure version it auto- 
matically creates a pull request in GitHuh, so tests run automaticaUy and I can deploy 
quickly’ 

- Erik Romijn, Django core dev and security reviewer for Two Scoops of Django 1.8 
Usefiil links for updates to Django itself. 

>- The ofRcial Django weblog at dj angoproj ect. com/weblog/ 

>- The oflicial django-announce mailing list at groups. google. com/forum/# ! forum/ 
dj ango-announce 

26.20 Prevent Clickjacking 

Clickjacking is when a malicious site tricks users to click on a concealed element of another site that 
they have loaded in a hidden frame or iframe. An example is a site with a false social media Togin’ 
button that is really a purchase button on another site. 

Django has instructions and components to prevent this from happening: 

>- docs.dj angoproj ect.com/en/1.ll/ref/clickjacking/ 

26.21 Guard Against XML Bombing With defusedxml 

Attacks against XML libraries are nothing new. For example, the amusingly titled but devastating 
‘Billion Laughs’ attack (http: / /en. wi ki pedi a . org/wi ki /Bi llion_laughs) was discovered 
in 2003. 


363 




Chapter 26: Security Best Practices 


Unfortunately, Python, like many other programming languages, doesht account for this or other 
venues of attack via XML. Furthermore, third-party Python lihraries such as Ixml are vulnerable to 
at least 4 well-known XML-hased attacks. For a list of Python and Python lihrary vulnerahilities see 
pypi. python.org/pypi /defusedxml#python-xml-libraries. 

Fortunately for us, Christian Fleimes created defusedxml, a Python lihrary designed to patch 
Pythons core XML lihraries and some of the third-party lihraries (including Ixml). 

For more Information, please read: 

>• pypi .python.org/pypi /defusedxml 

26.22 Explore Two-Factor Authentication 

Two-factor authentication (2FA) requires users to authenticate by combining two separate means 
of identification. 

For modern web applications, what that usually means is you enter your password as well as a value 
provided to you on your mobile device. A value that is either sent to your personal phone number or 
is reset every thirty seconds. 

The advantage of 2FA is that it adds another component, one that changes frequently, to the authen¬ 
tication process, great for any site involving personal identity, finance, or medical requirements. 

The downside is that the user needs to have a charged mobile device with access to a network in order 
to log in to your site, making it not so ideal for users who may not have a charged mobile device or 
easy access to a network. 

>- en.wikipedi a.org/wiki/Two_factor_authentication 
>• pypi .python.org/pypi /django-two-factor-auth 


PACKAGE TIP: Look for TOTP in 2FA Products and Packages 


TOTP is short for en.wikipedia.org/wiki/Time-based_One-time_Password_ 
Algori thm, which is an open Standard used by Google Authenticator and many other Ser¬ 
vices. TOTP does not require network access, which is useful for building certain kinds of 
Django projects. However, SMS implementations of course require cellular network access 
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or third-party Services such as twi li o. com. 


WARNING: The Issue of 2FA Recovery 


An important issue is how people recover from loss of their 2FA token or phone number. 
Passwords are generally recovered by sending an e-mail with a secret link. However, if the 
2FA token can also be reset by e-mail, access to the users e-mail has basically become the 
single factor of authentication. Common methods include offering TOTP authentication 
with SMS as a fallback, or offering a number of recovery codes that need to be kept by the 
User. In some cases, organisations will only reset these tokens after receiving a scan of an 
identity card belonging to the account holder. In any case, a recovery process wiU be needed, 
so think of this in advance. 


26.23 Embrace SecurityMiddleware 

WeVe mentioned Djangos built-in django.middleware.security.SecurityMiddleware 
several times aiready in this chapter. We owe it to ourselves and our users to embrace and use this 
feature of Django. 

26.24 Force the Use of Strong Passwords 

A strong password is one that more than just a list of characters. It is long and preferably complex, 
including punctuation, digits, and both character cases. Let’s pledge to protect our users by enforcing 
the use of such passwords. 

So what makes the best password? 

Our opinion is that at this point in time, length is more important than complexity. An 8 character 
length password mixing cases, numbers, and special characters is easier by several orders of magnitude 
to break than a 50-character sentence of just lower cased letters. Of course, what’s even better is if 
you have a 30-50 character sentence that includes numbers, mixed cases, and special characters. 


Quality Password Specification 


Bad 6-10 characters of just the alphabet 
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Quality Password Specification 


Okay Minimum 8 characters, mixed case + numeric + special characters 

Better Minimum 30 characters of just the alphabet 

Best Minimum 30 characters, mixed case + numeric + special characters 

Table 26.1: Password Strength: Length vs Complexity 


Reference: xkcd.com/936/ 

26.25 Give Your Site a Security Checkup 

There are a number of Services that provide automated checkups for sites. They arent security audits, 
but they are great, free ways to make certain that your production deployment doesnt have any gaping 
security holes. 

pyup. io’s Safety library (github. com/pyupio/saf ety) checks your instaUed dependencies for 
known security vulnerabilities. By default it uses the open Python vulnerability database Safety DB, 
but can be upgraded to use pyup. i o’s Safety API using the --key option. 

Erik Romijn, on the Django security team, has created Pony Checkup (ponycheckup.com), an 
automated security checkup tool for Django websites. There are several security practices that can 
easily be probed from the outside, and this is what his site checks for. 

Mozilla also provides a similar, but non-Django specific Service caUed Observatory (observatory . 
mozi lla. org). 


26.26 Put Up a Vulnerability Reporting Page 

It’s a good idea to publish information on your site about how users can report security vulnerabilities 
to you. 

GitHubs “Responsible Disclosure of Security Vulnerabilities” page is a good example of this and 
rewards reporters of issues by publishing their names: 

help.github.com/articies/responsible-disclosure-of-security-vulnerabiliti es/ 
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26.27 Never Display S equential Primary Keys 

Displaying sequential primary keys is to be avoided because: 

>- They inform potential rivals or hackers of your volume 

>- By displaying thesevalueswe make it trivial to exploit InsecureDl rectObj ectRef erences 
>- We also provide targets forXSS attacks 

Here are some patterns for looking up records without revealing sequential identifiers: 


26.27.1 Lookup by Slug 

In the Django world, this is incredibly common. There are literally hundreds of examples avatlable 
on how to do it. This is the goto method for many Django projects. However, it becomes a little 
challenging when you have issues with duplicate slugs. In which case, one of the other methods 
apply. 


26.27.2 UUIDs 


Django comes with very useful models. UUIDFl eld. WhUe a use case for using them as primary 
keys for large distributed systems exists, they also serve nicely for public lookups. Here is a sample 
model: 
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return str(self.pk) 


And here is how we call that model: 



WARNING: The Dangers of Obfuscating Sequential IDs 


Slugs and UUIDs both have their disadvantages. The slug-based approach runs into coUi- 
sions quickly, causing things like, “vanilla”, “vanilla-2”, “vanilla-3” to occur. UUIDs, to put 
it simply, are long and not memorizable by most humans. What can we do? 

You can obfuscate the sequential ID. But we don’t recommend it. Why not? 

The short answer: Obfuscating is not an effective way to hide sequential IDs. 

The long answer: There are any number of methods for obfuscating numbers ranging from 
base64 encoding to using the hashids library. These approaches workby converting a 
number to a alphanumeric code and back again. They not oniy hide the number, they 
also shorten it. Sounds great, right? 

The problem is that every method of obfuscating sequential IDs is fundamentally 
insecure. Base64 encoding is trivial to undo. Libraries like hashids can be broken 
with brute- forceattacks or by anyone with a good understanding of cryographic 
knowledge( carnage.github.io/2015/08/cryptanalysis-of-hashids). 

To summarize: If you want to hide your sequential identifiers, don’t rely on obfuscation. 


26.28 Reference Our Security Settings Appendix 

Keeping track of everything that relates to security and Django is challenging. This chapter alone is 
nigh 30 pages long and at the beginning we make it very ciear this is not an absolute reference. 
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26.29: Review the List of Security Packages 


In order to add clarity, weVe created Appendix G: Security Settings Reference. This is where we 
put important and useful information on how to better configure the security settings of a Django 
project. 


26.29 Review the List of S ecurity Packages 

In the security section of Appendix A: Packages, we list over ten related security packages that can 
make a dilference to your site. While some are listed in this chapter, others are unique to that section 
of this book. 


26.30 Keep Up-to-Date on General Security Practices 

We end this chapter with some common-sense advice. 

First, keep in mind that security practices are constantly evolving, both in the Django community 
andbeyond. Subscribe to groups. google. com/forum/# ! forum/d j ango-announce and check 
Twitter, Hacker News, and various security blogs regularly. 

Second, remember that security best practices extend weU beyond those practices specific to Django. 
You should research the security issues of every part of your web application stack, and you should 
foUow the corresponding sources to stay up to date. 


TIP: Good Books and Articles on Security 


Paul McMiUan, Django core developer, security expert, and Two Scoops reviewer, recom- 
mends the foUowing books: 

>- ‘‘‘‘The Tangled Web: A Guide to Securing Modern Web Applications' 
amzn.to/lhXAAyx 

>■ ‘‘‘‘The Web Application Hackers Handbooli 
amzn.to/ldZ7xEY 

In addition, we recommend the following reference site: 

>- wiki .mozilla.org/WebAppSec/Secure_Coding_Guidelines 
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26.31 Summary 

Please use this chapter as a starting point for Django security, not the ultimate reference guide. See 
the Django documentations list for additional security topics: 

docs.dj angoproj ect.com/en/1.11/topics/security/#additional-security-topics 

Django comes with a good security record due to the diligence of its community and attention to 
detail. Security is one of those areas where it’s a particularly good idea to ask for help. If you find 
yourself confused ahout anything, ask questions and turn to others in the Django community for 
help. 


370 




27 


Logging: What s It For, Anyway? 


Logging is like rocky road ice cream. Either you can’t live without it, or you forget about it and 
wonder once in awhile why it exists. 

Anyone who’s ever worked on a large production project with intense demands understands the im- 
portance of using the different log levels appropriately, creating module-specific loggers, meticulously 
logging information about important events, and including extra detail about the applications state 
when those events are logged. 

While logging might not seem glamorous, remember that it is one of the secrets to building extremely 
stable, robust web applications that scale and handle unusual loads gracefully. Logging can be used 
not only to debug application errors, but also to track interesting performance metrics. 

Logging unusual activity and checking logs regularly is also important for ensuring the security of 
your server. In the previous chapter, we covered the importance of checking your server access and 
error logs regularly. Keep in mind that application logs can be used in similar ways, whether to track 
failed login attempts or unusual application-level activity. 

27.1 Application Logs vs. Other Logs 

This chapter focuses on application logs. Any log file containing data logged from your Python web 
application is considered an application log. 

In addition to your application logs, you should be aware that there are other types of logs, and that 
using and checking all of your server logs is necessary. Your server logs, database logs, network logs, 
etc. all provide vital insight into your production system, so consider them all equally important. 
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27.2 Why Bother W ith Logging? 

Logging is your go-to tool in situations where a stack trace and existing debugging tools arent enough. 
Whenever you have different moving parts interacting with each other or the possibility of unpre- 
dictable situations, logging gives you insight into whats going on. 

The different log levels available to you are DEBUG, INFO, WARNING, ERROR, and CRITICAE. Let’s 
now explore when it’s appropriate to use each logging level. 


27.3 When to Use Each Log Level 


In places other than your production environment, you might as well use all the log levels. Log levels 
are controUed in your project’s settings modules, so we can fine tune this recommendation as needed 
to account for load testing and large scale user tests. 


In your production environment, we recommend using every log level except for DEBUG. 
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Figure 27.1: Appropriate usage of CRITICAL/ERRORAVARNING/INFO logging in ice cream. 


Since the same CRITICAE, ERROR, WARNING, and INFO logs are captured whether in production or 
development, introspection of buggy code requires less modification of code. This is important to 
remember, as debug code added by developers working to fix one problem can create new ones. 


The rest of this section covers how each log level is used. 
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27.3.1 Log Catastrophes With CRITICAL 

Use the CRITICAL log level only when something catastrophic occurs that requires urgent attention. 

For example, if your code relies on an internal web Service being available, and if that web Service is 
part ofyour sites core functionality, then you might log at the CRITICAL level anytime that the web 
Service is inaccessible. 

This log level is never used in core Django code, but you should certainly use it in your code anywhere 
that an extremely serious problem can occur. 


27.3.2 Log Production Errors With ERROR 

Let’s look at core Django for an example ofwhen ERROR level logging is appropriate. In core Django, 
the ERROR log level is used very sparingly. There is one very important place where it is used: whenever 
code raises an exception that is not caught, the event gets logged by Django using the following code: 



Howdoes Django put this to good use? WeU, when DE BUG= False is in your settings, everyone listed 
in ADMINS immediately gets emailed the following: 

>- A description of the error 

>- A complete Python traceback from where the error occurred 
>• Information about the HTTP request that caused the error 
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If youVe ever received one of those email notifications, you know how usefiil ERROR logs are when 
you need them most. 

Similarly, we recommend that you use the ERROR log level whenever you need to log an error that is 
worthy of being emailed to you or your site admins. When your code catches the exception, log as 
much information as you can to he ahle to resolve the prohlem. 

For example, an exception may he thrown when one of your views cannot access a needed third-party 
API. When the exception is caught, you can log a helpful message and the APFs failure response, if 
any. 


27.3.3 Log Lower-Priority Problems With WARNING 

This level is good for logging events that are unusual and potentiaUy had, hut not as had as ERROR- 
level events. 

For example, if you are using django-admin-honeypot to set up a fake admin/ login form, you might 
want to log intruders’ login attempts to this level. 

Django uses the log level in several parts of CsrfViewMiddleware, to log events that resuit in a 
403 Forbidden error. For example, when an incoming POST request is missing its csrf_token, 
the event gets logged as foUows: 
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27.3.4 Log Useful State Information With INFO 

We recommend using this level to log any details that may be particularly important when analysis 
is needed. These include: 

>- Startup and shutdown of important components not logged elsewhere 
>- State changes that occur in response to important events 
>- Changes to permissions, e.g. when users are granted admin access 

In addition to this, the INFO level is great for logging any general information that may help in 
performance analysis. It’s a good level to use while hunting down problematic bottlenecks in your 
application and doing profiling. 


27.3.5 Log Debug-Related Messages to DEBUG 


In development, we recommend using DEBUG and occasionally INFO level logging wherever youd 
consider throwing a pri nt statement into your code for debugging purposes. 

Getting used to logging this way isn’t hard. Instead of this: 


Example 27.3: Using Print to Display Data 

from django.views.generic import TemplateView 

from .helpers import pint_counter 

class Pint\/iew(TemplateView) : 

def get_context_data(self, *args, **kwargs); 

context = super(PintView, self).get_context_data(**kwargs) 
pints_remaining = pint_counter() 

print('Only %d pints of ice cream left.' % (pints_remaining)) 
return context 


We do this: 
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Sprinkling pri nt statements across your projects results in problems and technical debt: 

>- Depending on the web server, a forgotten print statement can bring your site down. 

>- Print statements are not recorded. If you dont see them, then you miss what they were trying to 
say. 

>• As the Django world migrates more and more to Python 3, old-style print statements like 
print IceCream.objects. flavorO willbreakyour code. 

Unlike print statements, logging aUows different report levels and different response methods. This 
means that: 

>- We can write DEBUG level statements, leave them in our code, and never have to worry about 
them doing anything when we move code to production. 

>- The response method can provide the response as email, log files, console and stdout. It can 
even report as pushed HTTP requests to applications such as Sentry\ 

Note that there’s no need to go overboard with debug-level logging. It’s great to add log¬ 
ging. debugO statements while youre debugging, but there’s no need to clutter your code with 
logging every single line. 
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Figure 27.2: Appropriate usage of DEBUG logging in ice cream. 


27.4 Log Tracebacks When Catching Exceptions 

Whenever you log an exception, it’s usually helpful to log the stack trace of the exception. Pythons 
logging module supports this: 

O Logger. exception () automatically includes the traceback and logs at ERROR level. 

0 For other log levels, use the optional exc_i nf o keyword argument. 

Flere’s an example of adding a traceback to a WARNING level log message: 



377 















Chapter 27: Logging: What’s It For, Anyway? 


except requests.HTTPError as e: 
logger.exception(e) 

logger.debug( 'Could not get additional data', exc_i nfo=True) 
return None 
return r 


27.5 One Logger Per Module That Uses Logging 

Whenever you use logging in another module, dont import and reuse a logger from elsewhere. In- 
stead, define a new logger specific to the module like this: 



What this gives you is the ability to turn on and off only the specific loggers that you currently need. 
If youre running into a strange issue in production that you can’t replicate locally, you can temporarily 
turn on DEBUG logging for just the module related to the issue. Then, when you identify the problem, 
you can turn that logger back off in production. 

27.6 Log Locally to Rotating Files 

When you create a new Django project with sta rtp ro j ect, your default settings file is configured 
to email ERROR and higher log messages to whomever you list in ADMINS. This occurs via a handler 
called Admi nEmai IHandler that comes with Django. 

In addition to this, we recommend also writing logs of level INEO and higher to rotating log files 
on disk. On-disk log files are helpful in case the network goes down or emails can’t be sent to your 
admins for some reason. Log rotation keeps your logs from growing to fili your available disk space. 

A common way to set up log rotation is to use the UNIX logrotate utility with log¬ 
ging. handlers.WatchedFileHandler. 
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27.7: Other Logging Tips 


Note that if you are using a platform-as-a-service, you might not be able to set up rotating log files. 

In this case, you may need to use an external logging Service such as Loggly: loggly. com. 

27.7 Other Logging Tips 

>- Control the logging in settings files per the Django documentation on logging: docs. 
dj angoproj ect.com/en/1.11/topics/logglng/ 

>- While debugging, use the Python logger at DEBUG level. 

>- After running tests at DEBUG level, try running them at INEO and WARNING levels. The re- 
duction in information you see may help you identify upcoming deprecations for third-party 
libraries. 

>■ Don’t wait untU it’s too late to add logging. YouU be grateful for your logs if and when your 
site fails. 

>- You can do useful things with the emails you receive when ERROR or higher level events occur. 
For example, you can configure a PagerDuty (pagerduty. com) account to alert you and your 
team repeatedly until you ve taken action. 


PACKAGE TIP: Logutils Provides Useful Handlers 


The logutils package by Vinay Sajip comes with a number of very interesting logging han¬ 
dlers. Features include: 

>- Colorizing of console streams under Windows, Linux and Mac OS X. 

>- The ability to log to queues. Useful in situations where you want to queue up log mes- 
sages to a slow handler like SMTPHandler. 

>- Classes that aUow you to write unit tests for log messages. 

>■ An enhanced HTTPHandler that supports secure connections over ITTTPS. 

Some of the more basic features of logutils are so useful that they have been absorbed into 
the Python Standard library! 


27.8 Necessary Reading Material 

>- docs.dj angoproj ect.com/en/1.11/topics/logglng/ 
>- docs.python.org/3/11brary/logglng.html 
>• docs.python.org/3/11bra ry/logglng.conf1g.html 
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> docs.python.org/3/library/logging.handlers.html 
>- docs.python.org/3/howto/loggi ng-cookbook.html 


27.9 Usefiil Third-Party Tools 

>■ Sentry (sent ry. i o) aggregatas errors for you and is trusted by the authors, Dropbox, AirBnB, 
and host of other firms. Their product is open source, and they have an awesome history of sup- 
porting various developers with their open source eiforts. We eant recommend them enough. 

>- Opbeat (opbeat. com) tracks errors and performance issues in your app. They provide some 
of the functionality of Sentry, and also include performance monitering. Unlike Sentry, they 
do not contribute back to the greater good of the open source community. 

>■ loggly.com (loggly . com) simplifies log management and provides various query tools. 

27.10 Summary 

Django projects can easily take advantage of the rich logging functionality that comes with Python. 
Combine logging with handlers and analysis tools, and suddenly you have real power. You can use 
logging to help you improve the stability and performance of your projects. 

In the next chapter weTl discuss signals, which become much easier to follow, debug, and understand 
with the help of logging. 
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Signals: Use Cases and 
Avoidance Techniques 


The Short Answer: Use signals as a last resort. 

Hie Long Answer: Often when new Djangonauts first discover signals, they get signal-happy. They 
start sprinkling signals everywhere they can and feeling like real experts at Django. 

After coding this way for a while, projects start to turn into confusing, knotted hairballs that 
can’t be untangled. Signals are being dispatched everywhere and hopefully getting received 
somewhere, but at that point it’s hard to teU what exactly is going on. 

Many developers also confuse signals with asynchronous message queues such as what Celery 
and Django Channels provides. Make no mistake, signals are synchronous andblocking, and caU- 
ing performance-heavy processes via signals provides absolutely no benefit from a performance 
or scaling perspective. In fact, moving such processes unnecessarily to signals is considered code 
obfuscation. 

Signals can be useful, but they should be used as a last resort, only when theres no good way 
to avoid using them. 


28.1 When to Use and Avoid Signals 

Do not use signals when: 

>- The signal relates to one particular model and can be moved into one of that modefs methods, 
possibly called by save (). 
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> The signal can be replaced with a custom model manager method. 

>- The signal relates to a particular view and can be moved into that view. 

It might be okay to use signals when: 

>• Your signal receiver needs to make changes to more than one model. 

>- You want to dispatch the same signal from multiple apps and have them handled the same way 
by a common receiver. 

>- You want to invalidate a cache after a model save. 

>- You want to create hooks for a third-party installable apps interaction with the database. In 
some cases this can be a better approach than extending objects. 

>• You have an unusual scenario that needs a caUback, and there’s no other way to handle it besides 
using a signal. For example, you want to trigger something based on the save () or i ni t () 
of a third-party apps model. You can’t modify the third-party code and extending it might be 
impossible, so a signal provides a trigger for a callback. 


TIP: Aymeric Augustin Thoughts on Signals 


Django core developer Aymeric Augustin says: ‘T advise not to use signals as soon as a regular 
function call will do. Signals obfuscate control flow through inversion of control. They make 
it diflicult to discover what code will actually run. 

Use a signal only if the piece of code sending it has positively no way to determine what its 
receivers will be.” 


28.2 Signal Avoidance Techniques 

Let’s go over some scenarios where you can simplify your code and remove some of the signals that 
you don’t need. 


28.2.1 Using Custom Model Manager Methods Instead of Signals 

Let’s imagine that our site handles user-submitted ice cream-themed events, and each ice cream event 
goes through an approval process. These events are set with a status of “Unreviewed” upon creation. 
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The problem is that we want our site administrators to get an email for each event submission so they 
know to review and post things quickly. 

We could have done this with a signal, but unless we put in extra logic in the post_save () code, 
even administrator created events would generate emails. 

An easier way to handle this use case is to create a custom model manager method and use that in 
your views. This way, if an event is created by an administrator, they don’t have to go through the 
review process. 

Since a code example is worth a thousand words, here is how we would create such a method: 



Now that we have our custom manager with its custom manager method in place, let’s attach it to 
our model (which comes with a noti fy_admi ns () method: 
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from .managers import EventManager 

class Event(TimeStampedModel) : 

STATUS_UNREVIEWED, STATUS_REVIEWED = (0, 1) 

STATUS_CHOICES = ( 

(STATUS_UNREVIEWED, "Unreviewed") , 

(STATUS_REVIEWED, "Reviewed") , 

) 

title = models.CharField(max_length=100) 
start = models.DateTimeField() 
end = models.DateTimeField() 

status = models.IntegerField(choices=STATUS_CHOICES, 

default=STATUS_UNREVIEWED) 

creator = models.ForeignKey(settings.AUTH_USER_MODEL) 
objects = EventManager() 

def notify_admins (self ): 

# create the subject and message 

subject = "{user} submitted a new event!" .format( 
user= self. creator.get_full_name()) 
message = """TITLE: {title} 

START: {start} 

END: {end}" format(ti tle=self .title, start=self .start, 

end=self .end) 

# Send to the admins! 

mail_admins(subj ect=subj ect, 
message=message, 
fail_silently= False) 


Using this follows a similat pattern to using the User model. To generate an event, instead of calling 
create (), we caU a create_event () method. 
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Example 28.3: Using the Custom Manager Method 


>>> from django.contrib.auth import get_user_model 
>>> from django.utils import timezone 
>>> from events.models import Event 

>>> user = get_user_model().objects.get(username="audreyr") 
>>> now = timezone.now() 

>>> event = Event.objects.create_event( 

... title="International Ice Cream Tasting Competition", 

start=now, 
end=now, 
user=user 
) 


28.2.2 Validate Your Model Elsewhere 

If youre using a pre_save signal to trigger input cleanup for a specific model, trywriting a custom 
validator for your field(s) instead. 

If validating through a ModelForm, try overriding your models clean () method instead. 


28.2.3 Override Your Moders Save or Delete Method Instead 

If youre using pre_save and post_save signals to trigger logic that only applies to one particular 
model, you might not need those signals. You can often simply move the signal logic into your modefs 
save() method. 

The same applies to overriding delete () instead of using pre_delete and post_delete signals. 
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28.2.4 Use a Helper Function Instead of Signals 

We find this approach useful under two conditions: 

O Refactoring-. Once we realize that certain bits of code no longer need to be obfuscated as signals 
and want to refactor, the question of‘Where do we put the code that was in a signal?’ arises. If 
it doesnt belong in a model manager, custom validator, or overloaded model method, where 
does it belong? 

0 Architecture: Sometimes developers use signals because we feel the model has become too heavy- 
weight and we need a place for code. While Fat Models are a nice approach, we admit it’s not 
much fun to have to parse through a 500 or 2000 line chunk of code. 

This solution, suggested to us by Django core developer Aymeric Augustin, is to place the code in 
helper functions. If done right, this helps us write cleaner, more reusable code. 

One interesting thing about this approach is to test the transition out of signals. Simply follow these 
steps: 

O Write a test for the existing signal call. 

0 Write a test for the business logic of the existing signal call as if it were in a separate function. 
0 Write a helper function that duplicates the business logic of the signal, matching the assertions 
of the test written in the second step. 

O Run the tests. 

0 Call the helper function from the signal. 

O Run the tests again. 

0 Remove the signal and call the helper function from the appropriate location. 

© Run the tests again. 

© Rinse and repeat until done. 

This approach allows us to carefully remove the signal without breaking things. It also helps us identify 
when an existing signal is required for a specific process. 

28.3 Summary 

Signals are a powerful tool in any Django developers toolbox. However, they are easy to misuse and 
it’s good practice to delve into why and when to use them. 
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What About Tbose Random 
Utilities? 


29.1 Create a Core App for Your Utilities 

Sometimes we end up writing shared classes or little general-purpose Utilities that are useflil every- 
where. These bits and pieces dont belong in any particular app. We don’t just stick them into a 
sort-of-related random app, because we have a hard time finding them when we need them. We also 
dont like placing them as “random” modules in the root of the project. 

Our way of handling our Utilities is to place them into a Django app caUed core that contains modules 
which contains functions and objects for use across a project. (Other developers foUow a similar 
pattern and call this sort of app common, generic, ufil, or ufils.) 

For example, perhaps our project has both a custom model manager and a custom view mixin used 
by several different apps. Our core app would therefore look like: 


Example 29.1: Core App Layout Example 


core/ 

_i nit_.py 

managers.py # contains the custom model manager(s) 
models.py 

views.py # Contains the custom view mixin(s) 
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TIP: Always Make the Core App a Real Django App 


We always make the core directory a Django app. At some point we inevitably end up doing 
at least one of the foUowing: 

>- Have non-abstract models in core. 

>■ Have admin auto-discovery working in core. 

>■ Have template tags and filters in core. 


Now, if we want to import our custom model manager and/or view mixin , we import using the same 
pattern of imports we use for everything else: 



29.2 Optimize Apps With Utility Modules 

Synonymous with helpers, these are commonly caUed utils.py and sometimes helpers.py. They are 
places where we place functions and classes which make common patterns shorter and easier. Let’s 
go into why this is helpful. 


29.2.1 Storing Code Used in Many Places 

There are times when we have functions or classes used in several places that doesht quite fit in 
models.py,forms.py, or any other specifically named module. When this occurs, we put this logic in 
the utils.py module. 

29.2.2 Trimming Models 

This is best explained with an example. 

We use the Flavor model frequently. We start attaching field after field, method after method, 
property after property, classmethod after classmethod. One day we notice that our fat model has 
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reached brobdingnagian proprtions and is over a thousand lines of code. Debugging and maintenance 
have become hard. What do we do? 

We start looking for methods (or properties or classmethods) whose logic can be easily encapsulated 
in functions stored 'mjlavors/utils.py. The existing methods (or properties or classmethods) become 
simple wrappers calling functions iromflavors/utils.py. The resuit is a more distributed code base that 
encourages code reuse as well as easier testing. 


29.2.3 Easier Testing 

A side efFect of moving logic from more complex constructs into functions placed in isolated modules 
is that it becomes easier to test. By isolation we mean it is usually imported within the app, rather 
than doing in-app/in-project imports. This causes less business logic overhead, hence making it easier 
to write tests for what logic is present in the module. 


TIP: Make UtUity Code Constructs as Focused as Possible 


Be it a function or a class, avoid aUowing multiple behaviors or conditions. Each utility func- 
tion should do one and only one thing weU. Don’t repeat yourself Don’t create utility func¬ 
tions that are duplicates of model behaviors. 


29.3 Django’s Own Swiss Army Knife 

The Swiss army knife is a multi-purpose tool that is compact and useful. Django has a number of 
usefulhelperfunctions thatdont have abetterhorne than the django. uti Is package. It’s tempting 
to dig into the code in django. uti Is and start using things, but don’t. Most of those modules are 
designed for internal use and their behavior or inclusion can change between Django versions. 

Instead, read docs. dj angoproj ect. com/en/1.11/ ref/uti Is/ to see which modules in there 
are stable. 


TIP: Malcolm Tredinnick on Django’s Utils Package. 


Django core developer Malcolm Tredinnick liked to thinkof dj ango. uti Is as being in the 
same theme as Batmans utility belt: indispensable tools that are used everywhere internaUy. 
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There are some gems in there that have turned into best practices: 

29.3.1 django.contrib.humanize 

This is a set of localized template filters designed to give user presented data a more ‘human’ touch. 
For example it includes a filter called ‘i ntcomma’ that converts integers to strings containing commas 
(or periods depending on locale) every three digits. 

While dj ango. contri b. humani ze’s filters are useful for making template output more attractive, 
we can also import each filter individually as a function. This is quite handy when processing any sort 
of text, especially when used in conjunction with REST APIs. 


29.3.2 django.utils.decorators.method_decorator(decorator) 

Django has some reaUy great function decorators. Many of us have written decorators for Django 
projects, especially when we’re working with Function-Based Views. However, there comes a time 
when we discover that our favorite function decorator would also make sense as a method decorator. 
Fortunately, Django provides the method_decorator 


29.3.3 django.utils.decorators.decorator_from_middleware( 
middleware) 

Middleware is a wonderful tool, but is global in nature. This can generate extra queries or other 
complications. Fortunately, we can isolate the use of middleware on a per-view basis by using this 
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view decorator. 

Also see the related decorator_f rom_middleware_wi th_args decorator. 


29.3.4 django.utils.encoding.force_text(value) 


This forces Django to take anything and turn it into a plain str representation on Python 3 and 

uni code on Python2. It avoids the display of a django. uti Is . functional._proxy _ohject. 

For more details, see Appendix D. 


29.3.5 django.utils.functional.cached_property 


Reinout van Rees educated us about this incredibly useful method decorator introduced in Django 
1.5. What it does is cache inmemory the resuit of a method with a single self argument as aproperty. 
This has wonderful implications in regards to performance optimization of a project. We use it in 
every project, enjoying how it ahows us to cache the results of expensive computations trivially. 

For a description on how to use the cached_p rope rty decorator, the official Django documentation 
on the subject is excellent: docs. d j angoproj ect. com/en/1. ll/ref/utils/#dj ango. uti Is. 
functional.cached_property 

In addition to the potential performance benefits, weVe used this decorator to make sure that values 
fetched by methods remain static over the lifetime of their ohject. This has proven very useful when 
dealing with third-party APIs or dealing with database transactions. 


WARNING: Careful Using cached_property Outside of Django 


Itis temptingto copy/paste the source code for cached_p rope rty for use outside of Django. 
However, when used outside a web framework, we discovered the code for this function has 
problems in multithreaded environments. Therefore, if coding outside of Django, you might 
want to take a look at the third-party cached_property library: 

>- github.com/pydanny/cached-property 
>- pydanny.com/cached-property.html 
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29.3.6 django.utils.htinl.forinat_htinl(fonnat_str, 
args, **kwargs) 


This is similar to Pythons str . format () method, except designed for building up HTML frag- 
ments. All args and kwargs are escaped before being passed to str. format () which then combines 
the elements. 

Reference: docs.dj angoproj ect.com/en/1.ll/ref/utils/#dj ango.uti Is.html. 

format_html 


29.3.7 django.utils.html.strip_tags(value) 


When we need to accept content from users and have to strip out anything that could be HTML, 
this function removes those tags for we while keeping all the existing text between tags. 


WARNING: SecurityAdvisoryon strip_tags Safety 


When using the stri p_tags function, or the stri ptags template tage, make absolutely 
certain that the outputted content is not marked as safe. This especially applies if you have 
disabled automatic escaping in your templates. Reference: 
djangoproj ect.com/weblog/2014/mar/22/strip-tags-advisory/ 


29.3.8 django.utils.six 

Six is a Python 2 and 3 compatibility library by Benjamin Peterson. It’s bundled directly into Django 
(hence its inclusion in Djangos utils library), but we can also find it as an independent package for 
other projects. 

>- Six on PyPI: pypi . python. org/pypi /six 
>- Six documentation: pythonhosted.org/six 
>- Six repo on GitHub: githuh.com/henjaminp/six 

>- Sixin Django: githuh.com/dj ango/dj ango/hloh/master/dj ango/utiIs/si x.py 
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Figure 29.2: Six smooths over the differences between 2 and 3. 


29.3.9 django.utils.text.slugify(value) 


We recommend that whatever you do, don’t write your own version of the slugi f y () function, as 
any inconsistency from what Django does with this function will cause subtle yet nasty problems in 
our data. Instead, we use the same function that Django uses and slugi fy () consistently. 


It is possible to use d j ango. templates. defaultfi Iters. slugi fy () in our Python code, as 
this calls the function described here. Nevertheless, we like to use the function directly from Django’s 
utils directory, as it is a more appropriate import path. 


However we decide to import this function, we try to keep it consistent across a project as there is a 
use case for when it has to be replaced, as described in the next subsection. 
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29.3.10 Slugification and Languages Besides English 


Tomek Paczkowski points out that Django’s built-in slugi f y () function can cause problems with 
localization: 



Fortunately, you can use the allow_uni code flag to overcome this issue: 



PACKAGE TIP: awesome-slugify 


If you want even more control over slugification you can’t do wrong with the awesome-slugify 
package. It provides a greater control over the slugification process, allowing the following 
benefits: 

>- Customizing of seperators 

>- Detailed control of the case of the outputting string 
>- Translation mappings 

>- Is Django-independent, which is useful for microservices 
>- Lots more! 

References: 

>- github.com/dimka665/awesome-slugify 

>- pydanny. com/awesome-slugi fy-human- readable-url-slugs- f rom-any-str|i ng. 
html 
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29.3.11 django.utils.timezone 

It’s good practice for us to have time zone support enabled. Chances are that our users live in more 
than one time zone. 

When we use Djangos time zone support, date and time information is stored in the database uni- 
formly in UTC format and converted to local time zones as needed. 


29.3.12 django.utils.translation 

Much of the non-English speaking world appreciates use of this tool, as it provides Django’s il8n 
support. See Appendix D: Internationalization and Localization for a more in-depth reference. 

29.4 Exceptions 

Django comes with a lot of exceptions. Most of them are used internally, but a few of them stand 
out because the way they interact with Django can be leveraged in fun and Creative ways. These, and 
other built-in Django exceptions, are documented at docs.djangoproject.com/en/dev/ref/ 
exceptions. 


29.4.1 django.core.exceptions.ImproperlyConfigured 

The purpose of this module is to inform anyone attempting to run Django that there is a configuration 
issue. It serves as the single Django code component considered acceptable to import into Django 
settings modules. We discuss it in both Chapter 5: Settings and Requirements Files and Appendix 
E: : Settings Alternatives. 


29.4.2 django.core.exceptions.ObjectDoesNotExist 


This is the base Excepti on from which all DoesNotExist exceptions inherit from. Weve found 
this is a reaUy nice tool for working with utility functions that fetch generic model instances and do 
something with them. Here is a simple example: 
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Also using this exception, we can create our own variant of Django’s 
dj ango. shortcuts. get_obj ect_or_404 function, perhaps raising a HTTP 403 excep¬ 
tion instead of a 404: 
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29.4,3 django, core .exceptions, PermissionD enied 

This exception is used when users, authenticated or not, attempt to get responses from 
places they are not meant to be. Raising it in a view wiU trigger the view to return a 
django.http.HttpResponseForbidden. 

This exception can prove useful to use in functions that are touching the sensitive data and compo- 
nents of a high-security project. It means that if something bad happens, instead of just returning a 
500 exception, which may rightly alarm users, we simply provide a “Permission Denied” screen. 



In this case, if this function were called by a view and something was ‘not right,’ then the Permi s- 
si onDeni ed exception would force the view to display the project’s 403 error page. Speaking of 403 
error pages, we can set this to any view we want. In the root URLConf of a project, just add: 



As always, with exception-handling views, because they handle all HTTP methods equaUy, we prefer 
to use function-based views. 
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29.5 Serializers and Deserializers 

Whether it’s for creating data files or generating one-off simple REST APIs, Django has some great 
tools for working with serialization and deserialization of data of JSON, Python, YAML and XML 
data. They include the capahility to turn model instances into serialized data and then return it hack 
to model instances. 

Here is how we serialize data: 



Here is how we deserialize data: 
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from favorites.models import Favorite 

# Get and instantiate the serializer class 

# The 'json' can be repTaced with 'python' or 'xml'. 

# If you have pyyaml installed, you can replace it with 

# 'pyyaml' 

TSONSerializer = get_serializer( 'json' ) 
serializer = TSONSerializer() 

# open the serialized data file 
with open (' data.txt' ) as f: 

seri alized_data = f.read() 

# deserialize model data into a generator object 

# we'll call 'python data' 

python_data = seri alizer . deseri alize(seri alized_data) 

# iterate through the python_data 
for element in python_data: 

# Prints 'django.core.seri alizers.base.DeserializedObject' 
print(type (element)) 

# Elements have an 'object' that are literally instantiated 

# model instances (in this case, favorites.models.Favori te) 

pri nt ( 

element . object . pk, 
element . obj ect . created 

) 


Django already provides a command-line tool for using these serializers and deserializers: the dump- 
data and loaddata management commands. While we can use them, they don’t grant us the same 
amount of control that direct code access to the serializers provides. 

This brings us to something that we always need to keep in mind when using Djangos huilt-in 
serializers and deserializers: they can cause problems. From painful experience, we know that they 
dont handle complex data structures well. 

Consider these guidelines that we follow in our projects: 
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>■ Serialize data at the simplest level. 

>- Any database schema change may invalidate the serialized data. 

>• Don’t just import serialized data. Consider using Django’s form libraries or Django Rest Frame- 
work serializers to validate incoming data before saving to the database. 

Let’s go over some of the features provided by Django when working with specific formats: 


29.5.1 django.core.serializers.json.DjangoJSONEncoder 


Out of the box, Pythons built-in JSON module can’t handle encoding of date/time or decimal types. 
Anyone who has done Django for a while has run into this problem. Fortunately for all of us, Django 
provides a very useflil 3S0NEncoder class. See the code example below: 



29.5.2 django.core.serializers.pyyaml 


While powered by the third-party library, pyyaml, Django’s YvAVIL serializer tools handles the time 
conversion from Python-toAAML that pyyaml doesnt. 
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For deserialization, it also uses the yaml. saf e_load () function under the hood, which means that 
we don’t have to worry about code injection. See Section 26.10.3: Third-Party Libraries That Can 
Execute Code for more details. 


29.5.3 django.core.serializers.xml_serializer 

By default Djangos XML serializer uses Pythons built-in XML handlers. It also incorporates ele- 
ments of Christian Heimes’ defusedxml library, protecting usage of it from XML bomb attacks. For 
more information, please read Section 26.21: Guard Against XML Bombing With defusedxml. 


29.5.4 rest_framework.serializers 

There are times when Django’s built-in serializers just don’t do enough. Here are common examples 
of their limitations: 

>- They only serialize data stored in fields. You can’t include data from methods or properties. 

>- You can’t constrain the fields serialized. This can be a security or performance consideration. 

When we run into these obstacles, it’s a good idea to consider switching to Django Rest Frameworlcs 
Serializers toolset. They allow for a lot more customization of both the serialization and deserializa¬ 
tion process. While that power comes with complexity, weVe found it’s worth it to use this tool rather 
than constructing a manual process from scratch. 

References: 

>- django- rest-framework.org/api-guide/seriali zers/ 

>■ Serializing Objects: 

django- rest-framework.org/api-guide/seriali zers/#serializing-obj ects 
>- Deserializing Objects: 

django- rest-framework.org/api-guide/seriali zers/ 

#deserializing-obj ects 
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29.6 Summary 

We follow the practice of putting often reused files into utility packages. We enjoy being able to 
remember where we placed our often reused code. Projects that contain a mix of core, common, util, 
and utils directories are just that much harder to navigate. 

Djangos own ‘utility belt’ includes a plethora of useful tools, including useful functions, exceptions, 
and serializers. Leveraging them is on of the ways experienced Django developers accelerate devel- 
opment and avoid some of the tangles that can be caused by some of the very features of Django. 

Now that weVe covered tools to make things work, in the next chapter we’ll begin to cover sharing 
a project with the world. 
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Deployment: Platforms as a 
Service 


Ifyoure working on a small side project or are a founder of a small startup, you’ll definitely save time 
by using a Platform as a Service (PaaS) instead of setting up your own servers. Even large projects 
can benefit from the advantages of using them. 

First, a public Service message: 


TIP: Never Get Locked Into a Platform as a Service 


There are amazing Services which wiU host your code, databases, media assets, and also pro¬ 
vide a lot of wonderful accessories Services. These Services, however, can go through changes 
that can destroy your project. These changes include crippling price increases, performance 
degradation, unacceptable terms of Service changes, untenable Service license agreements, 
sudden decreases in availability, or can simply go out of business. 

This means that it’s in your best interest to do your best to avoid being forced into architectural 
decisions based on the needs of your hosting provider. Be ready to be able to move from one 
provider to another without major restructuring of your project. 

We try to make sure that our projects are not intrinsically tied to any hosting solution, mean- 
ing that we are not locked into a single vendors pricing, policies, and functionality. 


As a WSGI-compliant framework, Django is supported on many PaaS providers. The most 
commonly-used Django-friendly PaaS companies as of this writing are: 


403 







Chapter 30: Deployment: Platforms as a Service 


>■ Elastic Beanstalk (aws. amazon. com/elasti cbeanstalk/) is an up-and-coming PaaS in 
the Python world. It comes with built-in autoscaling and tight integration with other AWS 
tools. 

>- Heroku (heroku . com) is a popular option in the Python community well known for its doc- 
umentation and add-ons System. 

>- PythonAnywhere (pythonanywhere.com) is a Python-powered PaaS that is incredihly 
beginner-friendly. 

Why do we like these Services? WeVe evaluated them carefuUy for our needs. Your needs may be 
different, so read on about how to choose a PaaS. 

30.1 Evaluating a PaaS 

When a PaaS is chosen to host a project, that project forces architecture concessions in order for the 
application to work in their system. Therefore, even if we heed our warning at the top of this chapter, 
extracting ourselves from the PaaS takes effort and time. 

Therefore, when a PaaS is chosen for a project, or while we are using a PaaS, we constantly consider 
the foUowing: 


30.1.1 Compliance 

Before you begin evaluating any other aspect, it’s critical to check to see if the PaaS meets local or 
federal mandates. Examples: 

>- Many medical-based projects in the United States require meeting HIP7\A standards. If the 
PaaS doesnt meet HIPAA standards, and the project contains user medical data and a project 
is deployed there, everyone involved is at risk for civil and criminal pro se cution under Title II of 
HIPAA. See en.wikipedia.org/wiki /HIPAA#Security_Rule 
>■ Most e-commerce projects require at least SSL, and anything dealing with credit cards 
needs to adhere to PCI. While Services like Stripe often make this moot, many projects 
require internal integration of credit card processing. Make sure the PaaS complies with 
the PCI specification. See en.wikipedia.org/wiki /Payment_Card_Industry_Data_ 
Security_Standard. 
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>■ For the European Union, if you are processing any identifiable data you’ll need to follow EU 
Directive 95/46/EC on the protection of personal data into account. Amongst other things, 
this places restrictions on exporting such data outside the European Union, especially if your 
PaaS doesnt participate in the US-EU Safe Harhor. If in douht, consuit legal counsel. 


30.1.2 Pricing 

Most PaaS options provide a free tier for heginner and toy projects, and Heroku and PythonAny- 
where are good examples of this trend. We ve gotten a lot of mileage out of this, and it’s heen great. 
You can even add extra Services for a reasonahle monthly fee. However, if one loses track of projects 
and Services, then this ‘reasonahle fee’ can quickly add up to a hefty monthly Service hili. Therefore, 
it’s a good idea to keep up on Service costs and your monthly provider hilis. 

At the other end of things, if high traflic is anticipated, it’s a good idea to see how much a site will cost 
with all the settings dialed up. For example, Heroku maxed out on dynos and enterprise PostgreSQL 
wiU cost over $40,000 a month. While the chances of a project needing this much horsepower is slim, 
the fact that Heroku offers this means that it can and does happen. 

While aU of this is going on, keep in mind that PaaS companies are under no legal or moral obligation 
to keep their prices or pricing methods static. In fact, developers we know of have built architecture 
for projects to take advantage of how billing is done by a PaaS, only to face crippling bilis when the 
said PaaS changes its terms. To make matters worse, because they had tied their internal infrastructure 
tightly to the billing structure of the PaaS, they lacked the option of quickly moving off. 


30.1.3 Uptime 

For PaaS this is a very tricky issue. They would reaUy like to provide 99.999999% uptime (sometimes 
referred to as the ‘nines’), but even with the best engineering, it’s not entirely under their control: 

>- Most of them, including Heroku and PythonAnywhere, rent space from vendors such as AWS 
and Rackspace. If those Services go down, then they go down. 

>- All of them are reliant on the physical infrastructure of the internet. Natural disasters or indus- 
trial accidents can bring everything to a halt. 
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Even if we ignore these factors, providing a PaaS infrastructure is a hard business. It’s more than 
standing up servers or Linux containers, it’s maintaining a billing system, customer-facing tools, 
customer contact systems, and a host of other systems. This volume ofwork, challenging in its own 
right, can conflict with the business of making sure our projects work and scale as we need them. 

That said, because it is integral to their business to provide consistent Service, they aim for as high 
a stability number as they can. In general most PaaS companies have pretty good uptime, slowly 
increasing over time as they make continual system improvements. Furthermore, the good companies 
provide status pages and publish formal reports about any outages or issues. Therefore, we don’t bother 
with reading outage reports that are over a few months old, as they are not indicative of the current 
engineering status of a company. 

However, if there are recent, multiple reports of outages, or a recent outage of an unacceptable dura- 
tion, we consider other PaaS options. 


WARNING: IfYou Need Very High Uptime 


It’s worth mentioning that for projects that are life-critical, i.e. people could die if they lack 
immediate access, then a PaaS is not the right solution. Instead, please use an infrastructure 
Service that provides a formal Service License Agreement. 


30.1.4 Staffing 


Yes, it’s important to know about the staffing level of a PaaS: 

>- If a PaaS lacks staff, then they can’t provide 24x7 engineering support, especially across holidays. 
No matter how enthusiastic a small shop is, and the deals they offer, they can’t ftx problems 
when their engineer is sleeping. 

>- Do they have the staff to answer emails and problem tickets? If their engineering staff is man- 
aging aU of these requests, when do they have time to maintain the system? 

We recommend testing out their level of support and responsiveness by filing a support ticket early 
on. Use this opportunity to ask a thoughtful question about something thats unclear in their docu- 
mentation, or get needed help from their staff. 
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30.1.5 Scaling 

How easy is it to scale up? If an e-commerce site is mentioned on CNN or on national television, 
can the site be dialed up quickly? 

On the flip side, how easy it is to scale back down? Sometimes a traflic spike is followed by slow 
periods and it should be easy to dial things back. 

Finally, can we automate this process? 

For reference, we like how easy Elastic Beanstalk makes it to handle autoscaling. 


30.1.6 HTTP Server 

Most Python-friendly PaaS use Nginx or their own similar systems to serve data, and ali of them can 
handle WSGI. What you have to look out for is that some PaaS only support WSGI, which makes 
it impossible to use Django Channels. 

For example, at the time of this books publication, Elastic Beanstalk uses Apache and mod_wsgi. 
This makes it impossible to use it with Django Channels. If Django Channels is a must for you, then 
dont use Elastic Beanstalk. 


30.1.7 Documentation 

In Chapter 23: Documentation: Be Obsessed we make it pretty ciear that we reaUy care about docu¬ 
mentation. While we readily admit to exploiting every channel we know to ask questions (see Chap¬ 
ter 34: Where and How to Ask Django Questions), we want the Services that we use to have good, 
maintained documentation. It’s important to have this as readily-found reference material, and it 
demonstrates that the PaaS in question is serious about what they do. 


TIP: Why We DonT Document How Each PaaS Works 


Every PaaS changes their API and documentation over time, some more rapidly than others. 
Since the Django PaaS space is stili evolving rapidly, specific PaaS commands and instruc- 
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tions are not listed here. We ask the reader to foUow the documentation listed on the PaaS 
provider site. 


30.1.8 Performance Degradation 

Sometimes a project that has heen running for a while under consistent load starts to slow down. 

When this occurs, it could he caused hy one or more prohlems. We use the foUowing workflow: 

O Check the project’s commit history for changes could have caused a performance degradation. 
There may even be a major bug hiding. 

0 Examine the project for undiscovered performance bottlenecks. See Chapter 24: Finding and 
Reducing Bottlenecks. 

© Ask the PaaS support team to look into the problem. They might have a quick answer for you. 

O The physical hardware that the project is running on might have a problem. The ‘cloud’ is 
actually hardware and hardware breaks or gets old. Start up a new project instance, port the 
data, and update the DNS records to match. Sometimes that resolves the issue. 

0 Ask the PaaS support team for further assistance. It doesht hurt to ask for help, especiaUy as 
a paying customer. 

If none of this works, consider running the project on another PaaS or your own servers. If it runs 

well in another environment, it might be time to move it olf. 


TIP: Free/BeginnerTiers Will Run Slowly 


The free tier of any PaaS is not going to run fast or handle any significant load. That takes 
resources that cost the PaaS money. Even with the hefty angel or VC funding in the tech 
industry, it’s just not going to happen. If the PaaS provides a free or inexpensive tier that 
handles very high loads, see the next section on ‘Company Stability’ 


30.1.9 Geography 

Consider the location of primary usage compared to the location of the PaaS. For example, if the 
majority of users are in China, then a PaaS that only serves from US-based data centers isn’t a good 
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option. Latency issues can cause clients and users to become quickly unhappy with a project. 


30.1.10 Company Stability 

A PaaS is an enormous undertaking. When done well, it requires a lot of overhead. Engineers, servers, 
customer support, account, and marketing are ali expensive business. Since the advent of PaaS So¬ 
lutions, weVe seen a number of them fail because of lack of sales, over expenditure of funds, and 
sheer exhaustion by overworked stalF. Fortunately, theyve aU provided a grace period during which 
projects were given time to move off, but it’s not realistic to count on that. 

Therefore, it behooves us to look at the pricing pians carefuUy. Once a PaaS is out of its beta or initial 
launch period, if there isnt a way to capture profitability, then using the PaaS is risky. 


30.2 Best Practices for Deploying to PaaS 

30.2.1 Aim for Identical Environments 

The holy grail of deployment is identical environments between development and production. How- 
ever, as soon as you decide to use a PaaS, this is no longer possible as the production system con- 
figuration is beyond your control. Nevertheless, the closer your can keep things identical between 
development and production the more maintainable your project will be. 

To aid developers working with their systems, aU PaaS provide some measure of specifics as to their 
hosting environment. With the advent of Docker, some go as far as providing distributions that 
closely mirror production. We cover this futher in Section 2.5: Optional: Identical Environments. 


30.2.2 Maintain a Staging Instance 

With automation often comes the ability to run staging instances of projects at a lower cost tier. This 
is a great place to test production deployments, not to mention a place to demo feature changes. 
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30.2.3 Automate All the Things! 

When it comes time to push an update to a production instance, it’s never a good idea to do all the 
steps manually. It’s simply too easy to make a mistake. Our solution is to use simple automation using 
one of the following tools: 

>- Makefiles are useful for simple projects. Their limited capability means we woht be tempted 
to make things too fancy. As soon as you need more power, it’s time to use something else. 
Something like Invoke as described in the next bullet. 

>- Invoke is the direct descendant of the venerable Fabric library. It is similar to Fabric, but is 
designed for running tasks locaUy rather than on a remote server. Tasks are defined in Python 
code, which aUows for a bit more complexity in task definitions (although it’s easy to take 
things too far). It has fuU support for Python 3.3 and up. 


30.2.4 Multiple Requirements Files in Multiple Environments 

Most PaaS limit themselves to only reading the root requirements.txt file. While it can be good to be 
constrained to identical environments everywhere, under some circumstances we just need different 
versions of Software in different places. For example, production might run Django 1.11.3, and we 
want staging to run Django 1.11.5. 

When this need arises, we fall back on our automation and the use of cascading pip files. For example, 
let’s say we want to deploy with the packages described in staging.txt instead of requirements.txt. To 
handle this, we might have a Makejile with this commands: 
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30.3: Summary 


30.2.5 Prepare for Disaster With Backups and Rollbacks 

Even with ali the precautions we take, sometimes deployments just blow up. Therefore, before any 
change is pushed a live site, we make certain for a particular PaaS we know how to: 

>- Restore databases and user-uploaded files from backups. 

>- Roll back to a previous code push. 

30.2.6 Keep Ejcternal Backups 

The great virtue of PaaS is that they abstract away many deployment and operational issues, allowing 
us to focus on writing our project. With that comes the risk that the PaaS might encounter their own 
problems. While most PaaS provide the capability to generate backups to their own systems, it’s a 
good idea to periodically run backups to external Services. This includes the databases and uploaded 
User files. 

Suggestions for storing the data include Dropbox, Crashplan, Amazon S3, and Rackspace Cloud 
Files, but there are many more. Which Service to choose should be based on architectura! decisions 
such as the location of the PaaS (For example, Elastic Beanstalk- and Heroku-based projects would 
use Amazon Services). 


30.3 Summary 

Platforms as a Service are a great way to expedite delivery of deployable projects. They allow for de- 
velopers to quickly harness significant resources that are maintained by specialized operations teams. 
On the other hand, they do come with a price tag and various limitations. Therefore, deciding to use 
a PaaS should be based per the project and skill set at hand, not out of personal preference. 

In addition, it’s a good idea to honor the practices we provide in this chapter, or to listen carefuUy to 
peers to determine what they do to best utilize these Services. 

In the next chapter, we cover the nuts and bolts of deployment at a high level. 
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31 


Deploying Django Projects 


Deployment of Django projects is an in-depth topic that could fili an entire book on its own. Here, 
we touch upon deployment at a high level. 

31.1 Single-Server for Small Projects 

Single-server is the quickest way to get a small Django project up onto a server. It’s also the cheapest 
Django deployment option. 

The obvious drawback is that your server will go down if your website URL gets featured on Hacker 
News or any popular blog. 

31.1.1 Should You Bother? 

Typically we dont bother with the single-server setup even for small projects, because using 
cooki ecutter-dj ango with Elastic Beanstalk or even Heroku is less work and gives us peace- 
of-mind in the event of traflic spikes. 

However, we highly recommend that you try setting up a single-server Django deployment in these 
situations: 

>- If you ve never done it before. It’s an extremely important learning experience. Doing it will 
give you a deeper understanding of how Python web applications work. 

>- If your Django project is more of a toy project or experiment. Websites with paying customers 
can’t alford to risk downtime, but the risk of a temporary spike is bearable to non-paying side 
projects. 
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>■ If youre certain that one server is adequate for your sites trafRc. For example, a Django site 
for your wedding guests will probably be fine on a single server. 


31.1.2 Example: QuickUbuntii + Gunicorn Setup 

Here’s an example of howwe could deploy a single-server Django project easily with the following 
components: 

>• An old computer or cheap cloud server 
>• Ubuntu Server OS 
>- PostgreSQL 
>- Virtualenv 
>- Gunicorn 

You can either use a computer that you have lying around your house, or you can use a cheap cloud 
server from a provider like DigitalOcean, Rackspace, or AWS. 

Typicaly, we start out by installing the latest LTS version of Ubuntu Server onto a cloud server. Cloud 
server providers often have readymade disk images that are installable with a click, making this trivial. 
But doing this manually by downloading an installer like ubuntu . com/server works fine too. 

We then install the Ubuntu packages needed. They can vary, but typically we end up installing at 
least these: 

For pip/virtualenv python-pip, python-vi rtualenv 

For PostgreSQL postgresql,postgresql-contrib,libpq-dev, python-dev 

Notice how Gunicorn and Django arent in that list. Whenever we can install a Python package 
rather than an Ubuntu package, we go with the Python package. It’ll almost always be more recent. 

Then we do all the server setup basies like updating packages and creating a user account for the 
project. 

At this point, it’s Django time. We clone the Django project repo into our user’s horne directory and 
create a virtualenv with the project’s Python package dependencies, including Gunicorn. We create 
a PostgreSQL database for the Django project and run python manage. py migrate. 
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Then we run the Django project in Gunicorn. As of this writing, this requires a simple 1-line com- 
mand. See: 

docs.dj angoproj ect.com/en/1.11/howto/deployment/wsgi/gunicorn/ 


At this point, we see the Django site running when we go to the server’s IP address in a web browser. 
Then we can configure the server hostname and point a domain name at that IP address. 


Of course, this is just a quick overview that leaves out many important details. Once you have a 
single-server setup working experimentally, you’U want to go back and read Chapter 26: Security 
Best Practices, and then secure your server and site. 


You’ll also outgrow the single-server setup pretty quickly. At that point, you may get fancier with 
your setup, e.g. adding nginx. Redis, and/or memcached, or setting up Gunicorn behind an nginx 
proxy. Eventually, youTl want to either sign up for a PaaS or move to a multi-server setup. 


TIP: LookUp the Specifics Online 


Our aim here is to give you a general idea and example of how the most minimal Django 
deployment might work, rather than a detailed how-to guide. 

Because commands and package names change quickly, and because this is not a tutorial 
book, we have not provided links to any particular setup instructions. But you should be able 
to find detailed instructions with a quick web search. 


31.2 Multi-Server for Medium to Large Projects 


Companies and growing startups who opt not to use a PaaS typically use a multi-server setup. Here 
is what a basic one might look like: 
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Figure 31.1: Example of a basic multi-server setup. 


This is what you need at the most basic level: 

>- DataBase server. TypicaUy PostgreSQL in our projects when we have the choice, though 
Eventbrite uses MySQL. 

>- WSGI application server. Typically uWSGI or Gunicorn with Nginx, or Apache with 
mod_wsgi. 

Additionally, we may also want one or more of the following: 

>- Static file server. If we want to do it ourselves, Nginx or Apache are fast at serving static files. 
However, CDNs such as Amazon CloudFront are relatively inexpensive at the basic level. 

>- Caching and/or asynchronous message queue server. This server might run Redis, Mem- 
cached or Varnish. 

>- Miscellaneous server. If our site performs any CPU-intensive tasks, or if tasks involve waiting 
for an external Service (e.g. the Twitter API) it can be convenient to offload them onto a server 
separate from your WSGI app server. 

By having specialized servers that each focus on one thing, they can be switched out, optimized, or 
changed in quantity to serve a project’s needs. 
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TIP: Using Redis for AU Ephemeral Data 


Redis has similar features to Memcached, but adds in the following: 

>- Authentication, which Memcached doesnt have out of the box. 

>- State is saved, so if a server is restarted the data doesnt go away. 

>- Additional data types means it can be used as an asynchronous message queue, in 
conjunction with tools like celery and rq. 


Finally, we also need to be able to manage processes on each server. We recommend in descending 
order of preference: 

O Supervisord 
O init Scripts 



31.2.1 Advanced Multi-Server Setup 

Here is an example of a much larger multi-server setup, complete with multiple servers of each type 
and load balancing: 
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Figure 31.3: Example of an advanced multi-server setup. 


Load balancers can be hardware- or software-based. Commonly-used examples include: 

>- Software-based: HAProxy, Varnish, Nginx 
>- Hardware-based: Foundry, Juniper, DNS load balancer 

>- Cloud-based: Amazon Elastic Load Balancer, Rackspace Cloud Load Balancer 















31.3: WSGIApplication Servers 


TIP: Horizontal vs. Vertical Scaling 


The above is an example of horizontal scaling, where more servers are added to handle load. 
Before scaling horizontally, it’s good to scale verticaUy by upgrading your servers’ hardware 
and maxing out the RAM on each server. Vertical scaling is relatively easy, since it’s just a 
matter of throwing money at the problem. 


TIP: Scaling Horizontally and Sessions 


When scaling horizontally, make sure that users don’t need sticky sessions. For example, if 
someone uploads a file to server 1, and then comes back thru the load balancer and lands on 
server 2, that shouldnt cause problems. Ways around this are storing uploaded media in a 
common shared drive or more commonly on cloud-based Systems such as Amazon S3. 


31.3 WSGI Application Servers 

Always deploy your Django projects with WSGI. 

Django 1.8’s startproject command, sets up a wsgi.py file for us. This file contains the default 
configuration for deploying our Django project to any WSGI server. For what it’s worth, the sample 
project templates we recommend in Chapter 3: How to Lay Out Django Projects, also includes a 
wsgi.py in its config/ directory. 

The most commonly-used WSGI deployment setups are: 

O uWSGI with Nginx. 

0 Gunicorn behind a Nginx proxy. 

© Apache with mod_wsgi. 
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Here’s a quick summary comparing the three setups. 


Setup _ Advantages _ Disadvantages 


uWSGI with Nginx 

Lots of great features and options. 
Extremely configurable. Said to 
be better performing than the 
other Setup options. 

Documentation stili 
growing. Not as time-tested 
as Apache. Not as 
beginner-friendly as the 

others. 

Gunicorn (sometimes with 
Nginx) 

Written in pure Python. 
Supposedly this option has 
slightly better memory usage, but 
your mileage may vary. 

Documentation is brief for 
nginx (but growing). Not as 
time-tested as Apache. 

Apache with mod_wsgi 

Has been around for a long time 
and is tried and tested. Very 
stable. Works on Windows. Lots 
of great documentation, to the 
point of being kind of 
overwhelming. 

Unless run in Elastic 

Beanstalk, doesnt work 

with environment variables. 
Apache configuration can 
get overly complex. Lots of 
crazy conf files. Doesnt 
work with Channels. 


Table 31.1: Gunicorn vs Apache vs uWSGI 


Theres a lot of debate over which option is faster. Don’t trust benchmarks blindly, as many of them 
are based on serving out tiny “Helio World” pages, which of course will have different performance 
from real web applications. 

Ultimately, though, all three choices are in use in various high volume Django sites around the world. 
Configuration of any high volume production server can be very difficult, and if a site is busy enough 
it’s worth investing time in learning one of these options very weU. 

The disadvantage of setting up our own web servers is the added overhead of extra sysadmin work. 
It’s like making ice cream from scratch rather than just buying and eating it. Sometimes we just want 
to buy ice cream so we can focus on the enjoyment of eating it. 
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31.4 Performance and Tuning: uWSGI and Gunicorn 


uWSGI and Gunicorn are very popular amongst Django developers looking to squeeze every last bit 
of performance out of their web servers. As of now, uWSGI is more configurable, but Gunicorn is 
very configurable too, and arguably easier to configure. 

Usefiil reading: 

>- uwsgi-docs.readthedocs.org 

>- docs.djangoproj ect.com/en/1.ll/howto/deployment/wsgi/uwsgi/ 

>- j ustcramer.com/2013/06/27/serving-python-web-applications/ David 

Cramer’s blog article arguing for using Nginx + UWSGI 
>- gunicorn.org 

>- cerebralmanifest.com/uwsgi - vs-gunicorn 


31.5 Stability and Ease of Setup: Gunicorn and Apache 

If you just want to get a Django site up and running fast, Gunicorn or Apache are your best bet. 
Apache used to be the easiest option, but Gunicorn has come a long way. These days, with Gunicorn 
and the default Django-provided wsgi.py file, the setup “just works” with zero or minimal debugging. 


31.6 Common Apache Gotchas 


WARNING: Do Not Use mod_python 


The official Django documentation explicitly warns against using mod_python. Django’s 
mod_python support was deprecated in Django 1.3. In Django 1.5, the mod_python request 
handler was removed from Django. 

Unfortunately, there are stiU many Online resources that talk about configuring Django with 
mod_python, causing many people confusion. Do not use mod_python. If using Apache, use 
mod_wsgi instead. 
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31.6.1 Apache and Environment Variables 


Outside of Elastic Beanstalk, Apache doesht workwith environment variables as described in Chap¬ 
ter 5: Settings and Requirements Files. You’ll need to do something like load a local configuration 
file for secret values into your settings module written in .ini, .cfg, .json, or .xml formats. Please read 
Section 5.4: When You Can’t Use Environment Variables. 


31.6.2 Apache and Virtiialenv 


Thanks to the hard work of Graham Dumpleton, getting Apache to work with virtualenv is a task 
that’s pretty straightforward: 

>- If using mod_wsgi 3.4 or newer and daemon mode, just add the following option to the WS- 
GIDaemonProcess directive: 

python-home=/some/path/to/root/of/virtualenv 
>• If using embedded mode: WSGIPythonHome /some/path/to/root/of/vi rtualenv 
>- If using mod_wsgi 3.3 or older and daemon mode, instead use the foUow- 
ing option to WSGIDaemonProcess where X.Y is the Python version: python- 
path=/some/path/to/root/of/virtualenv/lib/pythonX.Y 



Figure 31.4: How ice cream is deployed to cones and bowls. 
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31.7 Automated, Repeatable Deployments 


When we configure our servers, we really shouldnt be SSHing into our servers and typing in con- 
figuration commands from memory. It’s too easy to forget what weVe done. If servers configured 
this way go down and need to be recreated in an emergency, it’s almost impossible to set them up 
identically to what we had before. 


When you have a lot of moving parts, ali those pieces need to be re-creatable in the event of a problem. 
Problems can and wiU occur. Relying on you or your system administrators memory of how be or 
she set up everything a year ago is dangerous. 


Instead, our server setup should be automated and documented in a way that makes it trivial to 
recreate everything from scratch. In the readers case, you or your sysadmin should be able to set up 
everything without having to log into a single server manuaUy. 


Specifically, this means: 


>- We should be able to spin up and configure our entire server setup from scratch by running a 
command, then sitting back and watching as everything happens automatically. 

>- Even if it’s just a single command, it should be documented precisely. Imagine that someone 
just got hired by our company. On their first day of work, without knowledge of our web 
application or servers, he or she should be able to open our deployment.rst document and set 
up our production servers. 

>- Each time we run the command, there should be no dependency on pre-existing server state. 

>- Any Scripts should be idempotent, producing the same results no matter whether they are run 
for the first time or the hundredth time. 


In order to achieve all of the above, companies who don’t rely on a PaaS typically use one or more 
infrastructure automation and management tools. 
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Figure 31.5: Infrastmcture automation can be very powerfiil. 


WARNING: Caveat About the Rest of This Chapter 


Our challenge in writing about automated deployments is twofold: 

>- Django is a web development framework, not a deployment system. 

>- The fact is that everything we wrote before has gotten stale. Web application deploy¬ 
ment is a rapidly moving target. 

Therefore, we’re going to cover best practices at a very high level. 


31.7.1 A Rapidly Changing World 

As mentioned above, the world of automated deployments is evolving quickly. How quickly? Let’s 
look at configuration management from the perspective of Two Scoops of Django: 


Date Milestone Status of Config Management Tools 


Through2011 Ancient History’ Chef/Puppet preferred, CFEngine 

predates the iPhone 
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2012 

Work on Two Scoops commences 

Chef/Puppet preferred, Salt/Ansible stili 
very experimental 

Jan 2013 

Two Scoops of Django 1.5 Alpha 

Chef and Puppet stili strong, Salt/Ansible 
getting popular 

Mar 2013 

Two Scoops of Django 1.5 Final 

Docker open sourced. 

Jan 2014 

Two Scoops of Django 1.6 

Salt/Ansible are stable and popular, 
Chef/Puppet not so much, Docker stiU 
experimental, surges forward 

Mar 2015 

Two Scoops of Django 1.8 

Docker for identical environments, many 
use it for deployments, Salt/Ansible going 
strong, Chef/Puppet waning, Kubernetes 
is on the radar 

2016 

Two Scoops of Django 1.8 

Docker is mature. Docker Swarm, 
Kubernetes and Mesos become popular 
for managing containers, Chef/Puppet 
declining rapidly 

2017 

Two Scoops of Django 1.11 

Docker is mature. Docker Swarm and 
Mesos strong, but not as popular as 
Kubernetes. Salt/Ansible starting to 

wane. 

2019 

Projected Django 2.2 Release 

Tools we use today may be relegated to 
the same boring place as CFEngine 


Tahle 31.2: Timeline of DevOps Tools 

If anything, this timeline makes it ciear that it’s a good idea for us not to tie our projects to a specific 
configuration management tool. Unfortunately, as a project grows over time, moving from one of 
these tools to another is a challenging process. Hence, we recommend choosing the tool of choice 
very carefully. 
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31.8 Which Automation Tool Should Be Used? 

Because Python web application deployment is such a huge problem and pain point, the space has 
been flooded with tools attempting to solve it. We’re hearing lots of big promises from every tool, 
but at this point no particular tool has gone mainstream as The Easiest Way to Deploy, Self-Hosted. 

31.8.1 Too Much Corporate FlufF 

In the past few years weVe seen an vibrant ecosystem of companies dedicated to the issue of automat- 
ing deployments. There is a lot of money to be made. 

The resuit is that there are many good tools being built, but there is also a lot of corporate fluff to 
sift through. The creators of various tools each have their own corporate interests in mind, increasing 
with the amount of funding that they obtain from investors. Because a lot of money is involved, they 
each have their own marketing departments putting out a lot of convincing promises about what 
their tools have to offer. 

This isht a bad thing per se, but it can make it hard for us to determine what is the right tool for us. 

31.8.2 Do Your Own Research 

Great work is being done on great tools, but until these tools mature further, it’s hard to figure out 
whats actually great and whats just a corporate promise. The only way to figure out what you like is 
to try everything, and to say “No, thanks!” when the “Kool-Aid” is being offered. 

31.9 Current Infrastructure Automation Tools 

Among Django users, Docker, Kubernetes, Ansible, and SaltStack are the most popular tools for 
automating deployments. 

AU of these automation tools tend to be complex to set up and use, with a steep learning curve. That’s 
because theyre designed to manage not just one server, but thousands or more. 

Here is what these tools can perform at large scale: 

Remote execution: 
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>■ Installing packages via apt-get or other system package management tools on remote servers 

>- Running commands on remote servers. For example, running the virtualenv command with 
the -no-site-packages option on staging/production servers. 

>- Starting Services, and restarting them under certain conditions. For example, restarting an 
Nginxweb server when the site’s Nginx configuration file changes. 

>- When a command is executed remotely, logging and returning the response from the server. 

Configuration management: 

>- Creating or updating conf files for Services. For example, creating a pg_hha.conf file for a 
freshly installed PostgreSQL instance. 

>- Populating configuration values differently for different servers, hased on variahles like each 
servers particular IP address or OS-specific information. 

Orchestration and targeting: 

>- Controlling which servers a joh is sent to, and when it should be sent. 

>- Managing various components at a high level, creating pipelines to handle different workflows. 

>- Pushing jobs to servers from a master server, in push mode. 

>- Asking the master server what needs to occur, in pull mode. 

Docker, Ansible, SaltStack are pretty similar and can perform all of the above. Let’s explore what 

differentiates them: 


Tool _ Pros _ Cons 


Docker Fast deployments due to only applying 

deitas. Containerization approach. YAML 
config. Gigantic community. Open source. 
Many tooling options. 

SaltStack Primarily push mode. Blazing fast 

transport via Omq. YAML config. Lots of 
Salt States and examples online. Large 
community. Open source. Written in 
Python. 


Written in Go. Periodic API 
changes. 

Complexity can be overwhelming. 
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Tool_Pros_Cons 


Ansible Primarily push mode. Doesnt require Slower transport over SSH, but can 

daemons running on remote servers aside use Fireball Mode which sets up a 
from OpenSSH. Easy to learn. YAML temporary Omq daemon, 
config. Open source. Written in Python. 


Kubernetes Up and coming. Supported by Google and 
other big firms. Large Community. Open 
source. 


Pain points with implementation 
details not mentioned in the 
documentation. Understanding it 
at a high level is challenging. Don’t 
try using it until you have huge pain 
points with your current 
deployments. 


Table 31.3: Infrastructure Automation Tool Comparison 


TIP: What About Fabric and Invoke? 


Fabric and its Python 3-friendly successor Invoke are tools that allow you to execute remote 
commands. Smaller in scope than the above, it focuses on doing one thing well. It is fre- 
quently used in conjunction with aU of the above tools. 


The trend now seems to be Docker, SaltStack, 7\nsible, and for those who want to challenge them- 
selves, Kubernetes. They all use YvAVlL for configuraton. Since the latter two are written in Python, 
as a Python user it’s easy to dig into their source code. The reality of development is that whenever 
you rely on a tool for long enough at large scale, you end up hitting bugs or interesting edge cases. 
When this happens, you’ll be grateful that you can search the issue tracker, find others with the same 
problem, and look at or even modify source code if you need to. 


Keep in mind that things are evolving quickly. If you spend a lot of your time on devops, you need to 
read blogs, follow other operations engineers on Twitter, attend infrastructure-related meetups, and 
keep the pulse of new developments. 
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Figure 31.6: HopefuUy, one day someone will invent a one-button machine that deploys Django 
projects and makes ice cream. 


31.10 Other Resources 

The following are useful references for deploying projects. 

>• highperformancedjango.com 

>• fullstackpython.com/deployment.html 

31.11 Summary 

In this chapter we provided a very high level overview for deploying Django projects, including Ba¬ 
sic descriptions of single and multi-server setups. We also covered different the three most popular 
WSGI application servers. Finally, we compared infrastructure automation and configuration man- 
agement tools. 
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Continuous Integration 


Continuous integration (CI) is one of those things where, to explain the concept, we quote one of its 
originators: 

Continuous integration is a Software development practice where members of a team 
integrate their work frequently, usually each person integrates at least daily — leading 
to multiple integrations per day Each integration is verified by an automated build (in- 
cluding test) to detect integration errors as quickly as possible. Many teams find that 
this approach leads to significantly reduced integration problems and allows a team to 
develop cohesive Software more rapidly. 

— Martin Fowler, martinfowler.com/articles/continuousIntegration. 
html 


Here’s a typical development workflow when using continuous integration: 

O Developer writes code, runs local tests against it, then pushes the code to an instance of a code 
repository on GitHub or GitLab. This should happen at least once per day. 

0 The code repository informs an automation tool that a new commit is ready for integration. 

© Automation integrates the code into the project, building out the project. Any failures during 
the build process and the commit are rejected. 

O Automation runs developer-authored tests against the new build. Any failures of the tests and 
the commit is rejected. 

0 The developer is notified of success or the details of failure. Based on the report, the developer 
can mitigate the failures. If there are no failures, the developer celebrates and moves to the 
next task. 
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The advantages of this process are immediately ciear. Thanks to continuous integration, we have the 
following: 

>- Earlier warnings of bugs and breakdowns. 

>■ Deployment-breaking issues in the code are more frequently caught. 

>• Daily merges to the main tmnk mean that no one persons code dramaticaUy changes the code 
base. 

>- Immediate positive and negative feedback is readily available about the project. 

>- Automation tools that make this possible include a lot of metrics that make both developers 
and managers happy. 


32.1 Principies of Continuous Integration 

Now that weve gone over why continuous integration is great, let’s go over some key components 
when using this work process. This is our interpretation of principies explored Martin Fowler’s 
discussion of the topic at martinfowler.eom/articles/continuousIntegration.html# 
PracticesOfContinuousintegration 


32.1.1 Write Lots of Tests! 

One of the nice things about continuous integration is that it ties so well with everything we discuss 
in chapter 22, our chapter on testing. Without comprehensive tests, continuous integration simply 
lacks that kiUer punch. Sure, some people would argue that without tests, continuous integration 
is useful for checking if a deployment would succeed and keeps everyone on the same branch, but 
we think they are thinking from the perspective of statically-typed languages, where a successful 
compilation already provides significant guarantees that the Software at least starts. 


32.1.2 Keeping the Build Fast 

This is a tricky one. Your tests should arguably be running against the same database engine as your 
production machine. However, under certain circumstances, tests can take a minute or ten. Once a 
test suite takes that long, Continuous Integration stops being advantageous, and starts becoming a 
burden. 


432 




32.2: Toohfor Continuously Integrating Your Project 


It’s at this point that developers (including the authors) begin considering using SqliteS in-memory 
for tests. We’ll admit that weVe done it ourselves. Unfortunately, because SQLite3’s behaves signifi- 
cantly differently than PostgreSQL or MySQL, this can be a mistake. For example, field types are 
not constrained the same way. 

Here are a few tips for speeding up testing on large projects: 

>- Avoid fixtures. This is yet another reason why we advise against their use. 

>- Avoid TransactionTestCase exceptwhen absolutely necessary. 

>- Avoid heavyweight setUp () methods. 

>■ Write small, focused tests that run at lightning speed, plus a few larger integration-style tests. 
>- Learn how to optimize your database for testing. This is discussed in public forums like Stack 
Overflow: stackoverflow.com/a/9407940/93270 


32.2 Tools for Continuously Integrating Your Project 


Use the foUowing tools: 


32.2.1 Tox 

tox.readthedocs.io 


This is a generic virtualenv management and testing command-line tool that allows us to test our 
projects against multiple Python and Django versions with a single command at the shell. You can 
also test against multiple database engines. It’s how the authors and oodles of developers around the 
World check the compatibility of their code against different versions of Python. 

If that isn’t enough to convince you: 

>■ Tox checks that packages install correctly with different Python versions and interpreters. Check 
on Python 2.1, 3.5„ 3.6, PyPy and more ali in one go! 

>- Tox runs tests in each of the environments, configuring your test tool of choice. 

>- Tox can act “as a frontend to continuous integration servers, reducing boilerplate and merging 
CI and shell-based testing.” 
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i. 






Figure 32.1: It’s easy to get carried away. Pictured are two absolutely unnecessary Tox environments, 
just so we could get extra tastes. 


32.2.2 Jenkins 

jenkins-ci.org 


Jenkins is a extensible continuous integration engine used in private and open source efforts around 
the World. It is the Standard for automating the components of Continuous Integration, with a huge 
community and ecosystem around the tool. If an alternative to Jenkins is considered, it should be 
done so after careful consideration. 


32.3 Continuous Integration as a Service 

Jenkins is an awesome tool, but sometimes you want to have someone else do the work in regards 
to setting it up and serving it. There are various Services that provide automation tools powered 
by Jenkins or analogues. Some of these plug right into popular repo hosting sites like GitHub and 
GitLab, and most provide free repos for open source projects. Some of our favorites include: 


Service Python Versions Supported Link 


Travis-CI 


3.6, 3.5, 3.4, 3.3, 2.7, PyPy travis-ci .org 
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Service Python Versions Supported Link 


AppVeyor (Windows) 3.6,3.5,3.4,3.3,2.7 appveyor. com 

GitLab 3.6,3.5,3.4,3.3,2.7, PyPy gitlab.com 


Table 32.1: Continuous Integration Services 


32.3.1 Code Coverage as a Service 


When we use continuous integration through one of the above CI Services, what we dont get back is 
our code coverage. This causes problems with Section 22.7: The Game of Test Coverage. Fortunately 
for us, Services like codecov. io can generate coverage reports and the game can continue. 

32.4 Additional Resources 

>- en.wikipedia.org/wiki /Continuous_Integration 
>- jenkins-ci .org 

>- caktusgroup.com/blog/2010/03/08/django-and-hudson-ci-day-l/ 

>• ci.djangoproject.com 

>- docs.python-guide.org/en/latest/scenarios/ci / 


32.5 Summary 

Continuous integration has become a Standard for open source and private projects around the world. 
While there is the cost of doing work up front, the benefits of safer deployments and more robust 
projects easily outweigh the investment. Furthermore, there are enough resources and recipes that 
setting up continuous integration is faster than ever. 

One final note: even if tests are not written for a project, the practice of continual project building 
makes continuous integration worth the setup. 
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33 


The Art of Debugging 


Whether they are on a brand new shiny project or a legacy Django system thats a decade old, debug¬ 
ging just happens. This chapter provides useful tips we can do to make the debugging process better 
and easier. 


33.1 Debugging in Development 

These are common tricies and tools for debugging locally. 


33.1.1 Use django-debug-toolbar 

WeVe already mentioned this invaluable package repeatedly. It is arguably the easiest/fastest way to 
display various debug information about the current request/response cycle. If you want to know how 
fast your templates are rendering, what queries are being made, and what variables are being used, 
this is the tool. 

If you don’t have it set up and configured, stop everything else you are doing and add it to your project. 

>• pypi .python.org/pypi /django-debug-toolbar 
>• django-debug-toolbar.readthedocs.org 

33.1.2 Tbat Annoying CB V Error 

If you are using CBVs this is an error you might see in the console or a view test: 
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This is a bug that djangonauts of ali skill levels introduce into their codebases. 

The first thingwe do whenwesee Type Error:_init_() takes exactly 1 argument (2 

given) in the console is we check our urls.py modules. Chances are somewhere we are defining 
URLs where we forgot to add the as_vi ew () method to our CBV routing. 


Example of TypeError generating code: 


Example 33.2: EIow to Throw a CBV TypeError 

# Forgetting the 'as_view()' method 
url(r'''$', HomePageVi ew, name=' horne') , 


Example of fixed code: 
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33.1.3 Master the Python Debugger 

Also known as just PDB, this in essence provides an enhanced REPL for interacting with source code 
at breakpoints you spedfy. You can also step forward in the code to see how things change as the 
code is executed. In regards to Django, there tends to he three places PDB is used: 

O Inside of test cases. 

0 During an HTTP request against development breakpoints allow us to examine at our own 
pace the process of evaluating the request. 

© To debug management commands. 


WARNING: Before Deploying Check for PDB 


Having PDB breakpoints in code that reaches production is disastrous as it wiU stop comple- 
tion of user requests. Therefore, before deployment, search the code for ‘pdb’. You can also 
rely on tools like flakeS to automatically check for existence of pdb (and other problems). 


While PDB is useful, it becomes dramatically more powerful when extended by third-party packages 
such as ipdb. What ipdb does is add the ipython interface to the PDB interface, turning a handy 
tool into a something worth celebrating with ice cream. 

References: 

>- Pythons pdb documentation: docs.python.org/3/library/pdb.html 
>• Using PDB with Django: /mi ke. ti g. as/blog/2010/09/14/pdb/ 

Packages: 

>• IPDB: pypi .python.org/pypi/ipdb 

>- Using IPDB with pytest pypi . python .org/pypi /py test-i pdb 

33.1.4 Remember the Essentials for Form File Uploads 

Unless we consistently work on file uploads, there are two easily forgotten items that will cause file 
uploads to fail silently. This can be very frustrating, as code that fails silently is harder to debug. 
Anyway, should there be any problems with file uploads, check the following: 
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1. Does the <form> tag include an encoding type? 



2. Do the views handle request. FILES? In Function-Based Views? 



Or what about Class-Based Views? 
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TIP: Form-Based Class Based Generic Views 


If a view inherits from one of the following then we dont need to worry about re- 
quest. FILES in your view code. Django handles most of the work involved. 

>- django. VIews. generic.edit. FormMixin 
>- d j ango. VI ews. generi c. edit. FormVi ew 
>- django.VIews.generic.edit.CreateView 
>- django.VIews.generic.edit.UpdateView 
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In these examples we don’t provide the code for the store. handle_uploaded_fi le () method. 
We’re just demonstrating where we might place such a method caU. 


33.1.5 Lean on the Text Editor or IDE 

When using a Text Editor like Sublime Text, Textmates, Vim, Emacs, or many other choices, find 
Python and Django specific options or plugins and use them. Even if aU they do is highlight code 
and identify PEP-8 violations, that will help immensely. 

When using an IDE (Integrated Development Environment) like PyCharm, PyDev, WingIDE, 
Komodo, etc., then ali of the IDE’s capabilities with Python and Django should be embraced and 
used. That means use breakpoints and other advanced features. If we arent using the IDE to the 
fullest, then why are we bothering with the effort of setting up the IDE? 

Just make sure not to code to the IDE (or Text Editor). See subsection 1.7.2. 


TIP: What Is The Best IDE or Text Editor? 


“ Whateveryouprefer the most.'' 

Each of us is an individual. Whatever flavor of ice cream or source code editor we prefer the 
most is our own unique expression. The only right answer is the one chosen for ourselves. 

For what it is worth, as of 2015 Audrey prefers PyCharm and Atom and Daniel prefers 
Atom. 


33.2 Debugging Production Systems 

There are some bugs that turn up in production that seem impossible to duplicate in development. 
This happens when there are specific conditions that cannot be easily duplicated locally, including 
load conditions, third-party APIs, and the size of data. Debugging these problems can be really 
frustrating and time consuming. This section provides some tips on things that can be done to ease 
the pain. 
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33.2.1 Read the Logs the Easy Way 


The problem with diving into production log files is they can be so large they obfuscate the cause of 
errors. Instead, use an error aggregator like Sent ry to get a better view into what is going on in your 
application. 


33.2.2 Mirroring Production 


This concept is to mirror production in an environment that can be accessed by the maintainers for 
the sake of debugging. The growing popularity of modern deployment techniques (PaaS, devops, 
identical environments) in theory makes this easier to do. GeneraUy, when duplicating a production 
environment, the foUowing steps are taken: 

O Behind a firewaU or some other protective measure, set up a remote server identical to the 
production environment. 

0 Copy production data over, taking special care to remove Personally Identifying Informa¬ 
tion. By this, we mean anything that can be used to identify critical details of individual users, 
including email addresses. 

© Provide shell access to those who need it. 

Once these steps have been taken, try to replicate the reported bug. If the production mirror is 
completely inaccessible to anyone outside the product (i.e. behind a firewaU), you can even consider 
changing the settl ngs. DEBUG to True. 


WARNING: Take Special Care With User Data 


Ludvig Wadenstein notes that email addresses “might not seem like a big deal to keep around 
on your dev server, but aU it takes is one small mishap and you have sent emails to aU of your 
users.” 

The same goes for things Uke OAuth tokens or API keys to third-party Services like strl pe. 
com, Google accounts, or anything else involving authentication or access. 
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33.2.3 UserBasedExceptionMiddleware 

What if you could provide superusers with access to the setti ngs. DEBUG=True 500 error page 
in production? That would make debugging much easier, right? The problem, of course, is that hav- 
ing setti ngs. DEBUG=True in production is serious security problem. However, thanks to Simon 
Willison, co-creator of Django, there is a way to use this powerful debugging tool in production. 



WARNING: UserBasedExceptionMiddleware is a Security Concern 


An attacker that managed to gain access to a super user account might be able to get deeper 
into the System based on the technical 500 response alone. Please keep this in mind when 
implementing User Based Excepti onMiddleware. 


33.2.4 That Troublesome setti ngs. ALL0WED_H0STS Error 

I see that my old nemesis, setti ngs . ALL0WED_H0STS, has returned. 

- Daniel Roy Greenfeld 

The ALL0WED_H0STS setting is a list of strings representing the host/domain names that a Django 
site can serve. This is a wonderful security measure that defaults to ['localhost' , '127.0.0.1' , 
'[::!]'] when setti ngs . DEBUG is set to True. For most projects, this means that Django “just 
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Works” during development. However, as soon as setti ngs. DEBUG is set to False, then AL- 
LOWED_HOSTS defaults to an empty list. An empty ALLOWED_HOSTS will generate constant 500 
errors. Checking the logs will show that Suspi ci ousOperati on errors are being raised. 

So what is happening? 

O settings.DEBUGis False 

0 Django has nothing in ALL0WED_H0STS hence can’t match the host/domains named against 
what is being served. For example, trying to serve pages from example.com will generate an 
error because 'example.com’ in [] is False . 

© Django thinks that something suspicious is going on, and raises a Suspi ci ousOperati on 
error. 

Therefore, whenever a project is deployed for the first time and always returns a 500 error, check 
setti ngs . ALL0WED_H0STS. As for knowing what to set, here is a starting example: 



Reference: 

>- docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts 


33.3 Feature Flags 

An incredibly powerful technique, Feature Flags allow us to turn a project’s feature on or off via a 
web-based interface. 


TIP: Simon Willison’ Advice on Feature Flags 


Django project co-creator Simon Willison says: 

“Feature flags offer the best bang-for-your-buck of anything Fve ever added to 
a codebase.” 
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Let’s say we’re adding a new feature to our site, perhaps the ability to remotely control a robot that 
serves ice cream. Works great on our laptops, fine on a QA server. Then we push the change to 
production, thousands of users start to command the robot and it goes haywire. Utter disaster unfolds 
as chocolate ice cream is served to vanilla ice cream aficionados. We quickly shut everything down 
then spend days cleaning up the mess. However, the damage is done as ice cream eaters around the 
World decide not to use our ice cream serving robot. 

While a silly example, this kind of thing happens. Identical environments can help, but arent always 
the answer. For example, it’s not uncommon for users to discover a broken element of a new feature 
or bug fix that was missed in tests. Anyone who has pushed code to a production server experiences 
this from time to time. Of course, you can ask people to play around on a QA or staging server, but 
that isn’t the same as having them use the production site for real. What if we could allow a subset 
of real users (i.e. ‘beta users’) defined through an admin-style interface to interact with a new feature 
before turning it on for everyone? 

This is what feature fiags are aU about! 

If fact, production problems have been uncovered by having a smaller set of users more wiUing to try 
a new, possibly buggy feature. This can include stalf, friends of stalT, and friendly users wiUing to try 
out beta-level features before anyone else gets a chance. 

33.3.1 Feature Flag Packages 

The two most common feature flag packages for Django are django-gargoyle and django-waffle. 
They both support a similar feature set, though Gargoyle offers more options for building custom 
segments in exchange for more complexity. Either of them are very useful tools worth adding to 
projects. 

>- github.com/disqus/gargoyle 
>• github.com/j socol/dj ango-waffle 

33.3.2 Unit Testing Code AfFected by Feature Fiags 

One gotcha with feature fiags is running tests against code that are turned off by them. How do 
we know that our new feature is tested when the flag to run them is turned off? The answer to this 
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question is that our tests should cover both code paths, with feature flags on or off. To do this, we 
need to familiarize ourselves with how to turn a feature flag on or oif within the Django testing 
framework: 

>• gargoyle.readthedocs.io/en/latest/usage/index.html#testing-switches 
>- http://waffle.readthedocs.io/en/latest/testing/autornated.html# 
testing-automated 


33.4 Summary 

Stili eant figure out the problem? No worries, in the next chapter we provide some great tips for 
asking questions. 
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34 I Where and How to Ask 
Django Questions 


All developers get stuck at one point or another on something that’s impossible to figure out alone. 
When you get stuck, don’t give up! 


TIP: The Django Code of Conduct 


This chapter provides instructioris on how to interact with the Django community. To keep 
things civil, the Django Software Foundation has instituted a formal code of conduct. 

Djangos code of conduct applies to ali spaces managed hy the Django project and the Django 
Software Foundation. That includes IRC and aU the mailing lists under d j angoproj ect. 
com. Amongst other things, the Code of Conduct States participants should he welcoming, 
considerate and respectful. Harassment and other exclusionary hehaviour are not acceptable. 
The Django Software Foundation Code of Conduct committee deals with possible violations 
of the Code of Conduct. 

The Django Code of Conduct: d j angoproj ect. com/conduct/ 

If you believe someone is violating the Code of Conduct, whether it’s aimed at you or at 
someone else, please report this to the committee. Even if you arent sure - theyd rather have 
a few extra reports about events which turn out not to be a violation, than not know about 
something that was a violation. The committee will keep your identity confidential. 

Reporting Code of Conduct Violations: dj angoproj ect. com/conduct/ reportl ng/ 
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34.1 What to Do When YouVe Stuck 

Follow these steps to increase your chances of success: 

O Troubleshoot on your own as much as possible. For example, if youre having issues with a 
package that you just installed, make sure the package has been installed into your virtualenv 
properly, and that your virtualenv is active. 

0 Read through the documentation in detail, to make sure you didnt miss something. 

© See if someone else has had the same issue. Check Google, mailing lists, and StackOverflow. 

O Can’t find anything? Now ask on StackOverflow. Construet a tiny example that illustrates the 
problem. Be as descriptive as possible about your environment, the package version that you 
installed, and the steps that you took. 

0 Stili don’t get an answer after a couple of days? Try asking on the django-users mailing list or 
in IRC. 

34.2 How to Ask Great Django Questions in IRC 

IRC stands for Internet Relay Chat. There are channels like #python and #django on the Freenode 
IRC network, where you can meet other developers and get help. 

A warning to those who are new to IRC: sometimes when you ask a question in a busy IRC channel, 
you get ignored. Sometimes you even get troUed by cranky developers. Dont get discouraged or take 
it personaUy! 

The IRC #python and #django channels are run entirely by volunteers. You can and should help out 
and answer questions there too, whenever you have a few free minutes. 

O When you ask something in IRC, be sure that youVe already done your homework. Use it as 
a last resort for when StackOverflow doesnt suffice. 

0 Paste a relevant code snippet and traceback into gl st. gl thub. com (or another pastebin). 

© Ask your question with as much detail and context as possible. Paste the link to your code snip- 
pet/traceback. Be friendly and honest. 


TIP: Use a Pastebin! 


Don’t ever paste code longer than a few characters into IRC. Seriously, don’t do it. 
YouTl annoy people. Use a pastebin! 
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O When others offer advice or help, thank them graciously and make them feel appreciated. A 
litde gratitude goes alongway. Alot ofgratitude could make someones day. Think about how 
you would feel if you were volunteering to help for free. 

34.3 Feed Your Brain 

Fili up your ice cream bowl with these tasty tidbits of Django and Python information. They’ll help 
you keep up to date with the latest and greatest. 

O djangoproject. com/communi ty Django Project’s Feeds 

0 Subscribe to PyCoders Weekly and Python Weekly, two excellent methods for finding new 
articles and packages on the Python programming language. 

34.4 Insider Tip: Be Active in the Community 

The biggest secret to getting help when you need it is simple: be an active participant in the Python 
and Django communities. 

The more you help others, the more you get to know people in the community. The more you put in, 
the more you get back. 

34.4,1 9 Easy Ways to Participate 

O Attend Python and Django user group meetings. Join ali the local groups that you can find on 
wi ki. python. org/moi n/LocalUserGroups. Search meetup.com for Python and join all 
the groups near you. 

0 Attend Python and Django conferences in your region and country. Learn from the experts. 
Stay for the entire duration of the sprints and contribute to open source projects. YouU meet 
other developers and learn a lot. 

0 Contribute to open source Django packages and to Django itseff. Find issues and volunteer to 
help with them. File issues if you find bugs. 

O Join #python and #django on IRC Freenode and help out. 

0 Find and join other smaller niche Python IRC channels. Theres #pyladies, and there are also 
foreign-language Python IRC channels listed on www. python. org/communi ty/i rc/. 

© Answer Django questions on StackOverfiow. 
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O Meet other fellow Djangonauts on Twitter and Facebok. Be friendly and get to know everyone. 

© Join the Django group on Linkedin, comment on posts, and occasionally post things that are 
useful to others. 

O Volunteer for diversity efforts. Get involved with Django Girls and PyLadies and help make 
the Python community more welcoming to women. Rememher that there are many angles 
to diversity: something as small as helping with a PyCon in an underrepresented country can 
make a major difference. 


\ / / 



Figure 34.1: The ice cream eating help desk. 


34.5 Summary 

One of the strengths of Django is the human factor of the community hehind the framework. Assume 
a friendly, open stance when you need guidance and odds are the community will rise to the task of 
helping you. They won’t do your joh for you, but in general they will reach out and attempt to answer 
questions or point you in the right direction. 
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Closing Thoughts 


While weVe covered a lot of ground here in this fourth edition of the book. Yet this is just the tip 
of the ice cream cone. For starters we plan to write more technical books and grow our open source 
projects. 

As for Two Scoops of Django, if there is another edition, it won’t be out until Django 2.2 LTS is 
released. The reason is that Django 1.11 is a Long Term Support version of Django, meaning that 
the content in this book will remain current until at least April 2020. 

We’d genuinely love to hear from you, and so would the rest of the Django community. For specific 
bookcontent-relatedfeedback, we’re using GitHub issues to tracksubmissions and commentary from 
readers. Report any of the following at github.com/twoscoops/two-scoops-of-django-l. 
ll/i ssues:: 

>- Did you find any of the topics unclear or confusing? 

>• Any errors or omissions that we should know about? 

>■ What additional topics do you think we should cover in this edition? 

We hope that this has been a useful and worthwhile read for you. If you enjoyed reading this book, 
please teU others by writing a positive reviews. We need and appreciate your support. 

Cheers to your success with your Django projects! 

Daniel Roy Greenfeld and Audrey Roy Greenfeld 

>- pydanny.com/ audreyr.com/ twoscoopspress.com 
>- GitHub: @pydanny, @audreyr, and @twoscoopspress 
>• Twitter: @pydanny, @audreyr, and @twoscoopspress 
>- Facebook: facebook.com/twoscoopspress 
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Appendix A: Packages Mentioned In This 
Book 


This is a list of the third-party Python, Django, and front-end packages that weVe described or 
mentioned in this book. WeVe also snuck in a few really useful packages that we dont mention in 
the book but that we feel are extremely useful. 

As for the packages that we’re currently using in our own projects: the list has some overlap with this 
list but is always changing. Please don’t use this as the definitive list of what you should and should 
not be using. 


Core 

Django djangoproject.com 

The web framework for perfectionists with deadlines. 
django-debug-toolbar dj ango-debug-toolbar. readthedocs.org 
Display panels used for debugging Django HTML views. 
django-model-utils pypl .python.org/pypi/django-model-utlls 
Useful model Utilities including a time stamped model. 
ipdb pypl.python.org/pypl/Ipdb 
IPython-enabled pdb 

Pillow pypl .python.org/pypl /Pillow 

Friendly installer for the Python Imaging Library. 
pip plp-lnstaller.org 

Package installer for Python. Comes built-in with Python 3.4 or higher. 
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pipenv docs.pipenv.org 

Sacred Marriage of Pipfile, Pip, & Virtualenv. Mostly replaces virtualenvwrapper and adds a 
lot more features in the process. 

Sphinx sphinx-doc.org 

Documentation tool for Python projects. 
virtualenv virtualenv.org 

Virtual environments for Python. 

virtualenvwrapper doughellmann. com/proj ects/vi rtualenvwrapper 
Makes virtualenv better for Mac OS X and Linux! 
virtualenvwrapper-win pypi .python.org/pypi/virtualenvwrapper-win 
Makes virtualenv better for Windows! win 


Asynchronous 

celery celeryproject.org 
Distributed task queue. 
flower pypi.python.org/pypi/flower 

Tool for monitoring and management of Celery tasks. 
django-channels pypi.python.org/pypi/django-channels 

Official Django websockets interface, can also be used as task queue. 
rq pypi.python.org/pypi/rq 

RQjs a simple, lightweight, Hbrary for creating background jobs, and processing them. 
django-rq pypi.python.org/pypi/dj ango-rq 

A simple app that provides django integration for RQ_(Redis Queue). 
django-background-tasks gi thub. com/arteri a/dj ango-background-tasks 
Database backed asynchronous task queue. 

DataBase 

psycopg2 pypi. python.org/pypi /psycopg2 
PostgreSQL database adapter. 

django-maintenancemode github. com/shanx/django-mai ntenancemode 
AUows you to turn a site on and off with a management command. 
django-maintenancemode-2 gi thub. com/alsoi code/d j ango-mai ntenancemode-2 

A database-powered solution great for flipping portions of a site in and out of read-only mode. 
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Deployment 


Fabric pypi.python.org/pypi/Fabric 

Simple tool for remote execution and deployment. 

Invoke pypi .python.org/pypi/invoke 
Like Fabric, also works in Python 3. 

Supervisor supervisord.org 

Supervisord is a client/server system that allows its users to monitor and control a number of 
processes on UNIX-like operating systems. 


Forms 

django-crispy-forms dj ango-crispy-forms. readthedocs. io 

Rendering Controls for Django forms. Uses Twitter Bootstrap widgets by default, but skinnable. 

django-flopp)d'orms d j ango- f loppyf orms. readthedocs. i o 

Form field, widget, and layout that can work with django-crispy-forms. 

django-forms-builder A Django reusable app providing the ability for admin users to 
create their own forms within the admin interface. github.com/stephenmcd/ 
django-forms-builder 


Front-End 

JSCS j scs.info 

JavaScript code style linter. 
CSScomb csscomb.com 

Coding style formatter for CSS. 


Logging 

logutils pypi . python .org/pypi /logutiIs 
Adds useful handlers for logging. 

Sentry sentry.io 

Exceptional error aggregation, with an open source code base. 
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Project Templates 

CookiecutterDjango github.com/pydanny/cookiecutter-django 
The sample project layout detailed in chapter 3 of this book. 

Cookiecutter cookiecutter. readthedocs. io 

Not explicitly for Django, a command-line utility for creating project and app templates. It’s 
focused, heavily tested and weU documented. By one of the authors of this book. 

RESTAPIs 

django-rest-framework django-rest-fram ework.org 

The defacto REST package for Django. Exposes model and non-model resources as a RESTful 
API. 

django-jsonview gi thub. com/j socol/d j ango-j sonvi ew 

Provides a simple decorator that translates Python objects to JSON and makes sure decorated 
views will always return JSON. 

django-tastypie dj ango-tastypie. readthedocs. io 

Expose model and non-model resources as a RESTful API. 


Security 

bleach pypi.python.org/pypi/bleach 

An easy whitelist-based HTML-sanitizing tool. 
defusedxml pypi.python.org/pypi /defusedxml 

Must-have Python library if you are accepting XML from any foreign source. 
django-autoadmin pypi.python.org/pypi/django-autoadmin 

Automatic admin users for Django projects with autogenerated passwords, takes the headache 
out of providing secure access with auto-created sites, 
django-admin-honeypot pypi .python.org/pypi /django-admin-honeypot 

A fake Django admin login screen to notify admins of attempted unauthorized access. 
django-axes github.com/django-pci/dj ango-axes 

Keep track of failed login attempts in Django-powered sites, 
django-csp github.com/mozilla/dj ango-csp 
Adds Content Security Policy to Django. 

django-ratelimit-backend pypi. python. org/pypi /django-ratelimi t-backend 
Login rate-limiting at the auth backend level. 
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django-restricted-sessions gi thub. com/eri kr/dj ango- restri cted-sessi ons 

This third-party package lets you restrict sessions to an IP or an IP range and/or the user agent, 
django-secure pyp"i. python. org/pypi /django-secure 

Helps you lock down your site’s security using practices advocated by security specialists. Much 
ofits functionality has been subsumed by Djangos SecurityMiddleware class. 
django-two-factor-auth pypi .python.org/pypi /django-two-factor-auth 
Complete Two-Factor Authentication for Django. 
django-user-sessions pypi .python.org/pypi/django-usei —sessions 
Django sessions with a foreign key to the user. 
peep pypi.python.org/pypi/peep 

Uses only verified TLS to upload to PyPI protecting your credentials from theft. Has other 
useful features worth looking at. 

Twine pypi.python.org/pypi/twine 

Uses only verified TLS to upload to PyPI protecting your credentials from theft. Has other 
useful features worth looking at. 


Testing 

coverage coverage.readthedocs. io 

Checks how much of your code is covered with tests. 
django-test-plus gi thub. com/ revsys/dj ango- test-plus 

Useful additions to Djangos default TestCase, which the Two Scoops authors learned from 
the creator of this package and continue to use to this day. We’re happy to see this bundled up 
for ease of use. 

factoryboy pypi.python.org/pypi /f actory_boy 
A package that generates model test data, 
modelmommy pypi .python.org/pypi/model_mommy 
Another package that generates model test data, 
mock pypi.python.org/pypi/mock 

Not explicitly for Django, this allows you to replace parts of your system with mock objects. 
This project made its way into the Standard library as of Python 3.4. 
pytest pytest.org 

A mature full-featured Python testing tool that is very useful for Python and Django projects. 

pytest-django pytest-dj ango. readthedocs .io 

pytest-django is a plugin for py.test that provides a set of useful tools for testing Django appli- 
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cations and projects. 
tox tox.readthedocs.io 

A generic virtualenv management and test command line tool that aUows testing of projects 
against multiple Python version with a single command at the shell. 


User Registration 

django-allauth django-allauth. readthedocs. io 

General-purpose registration and authentication. Includes Email, Twitter, Facebook, GitHub, 
Google, and lots more. 

python-social-auth dj ango-soci al-auth. readthedocs. io 

Easy social authentication and registration for Twitter, Eacebook, GitHub, Google, and lots 
more. 

django-registration gi thub. com/ubernostrum/dj ango- regi strati on 
A simple, extensible user-registration app for Django 

Views 

django-braces dj ango-braces.readthedocs.io 

Drop-in mbdns that reaUy empower Djangos class-based views. 

django-vaniUa-views d j ango-vani lia-vi ews. org 

Simplifies Django’s generic class-based views by simplifying the inheritance chain. 

Time 

python-dateutil pypi. python. org/pypi /python-dateuti 1 
Provides powerful extensions to Pythons datetime module. 

pytz pypi.python.org/pypi/pytz 

Brings the Olson tz database into Python. This library allows accurate and cross platform 
timezone calculations. It also solves the issue of ambiguous times at the end of daylight saving 
time. Library Reference 

Miscellaneous 

awesome-slugily pypi .python.org/pypi/awesome-slugify 
A flexible slugify function. 
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dj-stripe pypi .python.org/pypi/dj-stripe 
Django + Stripe made easy. 

django-compressor django-compressor. readthedocs.io 

Compresses linked and inline JavaScript or CSS into a single cached file, 
django-extensions dj ango- extensi ons. readthedocs. i o 

Provides shell_plus management command and a lot of other Utilities, 
django-haystack github.com/dj ango-haystack/dj ango-haystack 
FuU-text search that worlcs with SOLR, Elasticsearch, and more, 
django-js-reverse gi thub. com/i error/dj ango-j s- reverse 
JavaScript uri handling for Django that doesht hurt. 
django-pipeline gi thub. com/j azzband/d j ango-pi peli ne 

Compression of CSS and JS. Use with cssmin and jsmin packages. 
django-htmlmin github.com/cobrateam/django-htmlmin 
HTML minifier for django. 

django-reversion gi thub. com/eti anen/dj ango- reversi on 

An extension to the Django weh framework that provides comprehensive version control facil- 
ities. 

django-watson github.com/etianen/django-watson 

FuU-text multi-table search appUcation for Django using SQL database features. 
envdir gi thub. com/j ezdez/envdi r A Python port of daemontools’ envdir. 
flakeS pypi.python.org/pypi/flakeS 

Checks code quality by using PyFlakes, pep8, and other tools. 
pathlib pypi . python. org/pypi /pathli b Object-Oriented filesystem paths being merged into 
Python as of release 3.4. 
pip-tools gi thub. com/nvi e/pi p- tools 

A set of tools to keep your pinned Python dependencies fresh. 
pyyaml pypi.python.org/pypi/PyYAML 
YAML parser and emitter for Python, 
requests docs.python-requests.org 

Easy-to-use HTTP library that replaces Pythons urllib2 library. 
silk github.com/mtford90/silk 

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and Stores 
HTTP requests and database queries before presenting them in a user interface for further 
inspection. 
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Appendix B: Troubleshooting 
Installation 


This appendix contains tips for troubleshooting common Django installation issues. 

Identifying the Issue 

Often, the issue is one of: 

>- That Django isn’t on your system path, or 
>- That youre running the wrong version of Django 

Run this at the command line: 


Example 35.1: Checking Your Django Version 


python -c "import django; print django.get_version()" 

If youre running Django 1.11, you should see the following output: 


Example 35.2: Django Version 


1.11 

Dont see the same output? Well, at least you now know your problem. Read on to find a solution. 
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Our Recommended Solutions 

There are all sorts of different ways to resolve Django installation issues (e.g. manually editing your 
PATH environment variable), but the following tips will help you fix your setup in a way that is 
consistent with what we describe in chapter on The Optimal Django Environment Setup. 


CheckYour Virtualenv Installation 

Is virtualenv installed properly on your computer? At the command line, try creating a test Virtual 
environment and activating it. 

If youre on a Mac or Linux system, verify that this works: 


Example 35.3: Checking Virtualenv on Mac or Linux 


$ virtualenv testenv 
$ source testenv/bin/activate 


If youre on Windows, verify that this works: 


Example 35.4: Checking Virtualenv on Windows 


C:\code\> virtualenv testenv 
C:\code\> testenv\Seripts\acti vate 


Your virtualenv should have been activated, and your command line prompt should now have the 
name of the virtualenv prepended to it. 

On Mac or Linux, this will look something like: 


Example 35.5: Virtualenv Prompt on Mac and Linux 


(testenv) $ 


On Windows, this will look something like: 


464 











Example 35.6: Virtualenv Prompt on Windows 


(testenv) > 

Did you run into any problems? If so, study the Virtualenv documentation (vi rtualenv . org) and 
fix your installation of Virtualenv. 

If not, then continue on. 

Checkif Your Virtualenv Has Django 1.11 Installed 

With your virtualenv activated, checkyour version of Django again: 



Ifyou stiU don’t see 1.11, then try using pip to install Django 1.11 into testenv: 



Did it Work? Check your version of Django again. If not, check that you have pip installed correctly 
as per the official documentation (pip-installer.org). 

Check for Other Problems 

Follow the instructions in the official Django docs for trouhleshooting prohlems related to running 
django-admin.py: 

docs.dj angoproj ect.com/en/1.11/faq/troubleshooting/ 
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Appendix C: Additional Resources 


For a list of learning resources that are specific to current versions of Python and Django, please use 
the following links. They provide tities, descriptions, and links for books, videos, and web articles 
that serve as tutorials and references. 

>- twoscoopspress.com/pages/current-dj ango-books 
>- twoscoopspress.com/pages/dj ango-tutori ais 

This rest of this appendix lists additional resources that we feel are timeless. Some of them might be 
for previous versions of Python or Django, but the concepts they present transcend version. 


Timeless Python and Django Material 

Books: 

High Performance Django 

amazon.com/High-Performance-Dj ango/dp/1508748128/ 
highperformancedj ango.com 

Written with a focus on scaling Django, this book espouses many good practices. Full of useful 
information and tricks, as well as questions in each section that force you to think about what 
you are doing. 

Two Scoops of Django: Best Practices for Django 1.8 (print) 

twoscoopspress.com/products/two-scoops-of-dj ango-1-8 
The third printed edition of this book. 


Web: 
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Django Packages 

dj angopackages.org 

A directory of reusable apps, sites, tools, and more for your Django projects maintained by the 
authors of Two Scoops of Django. 

Classy Class-Based Views 
ccbv .co.uk 

A website that has provides detailed descriptions, with full methods and attributes, for each of 
Djangos class-based generic views. 

Classy Django REST Framework 

cdrf.co 

Detailed descriptions, with fiill methods and attributes, for each of Django REST Framework’s 
class-based views and serializers. 

pydanny’s blog 

pydanny.com/tag/dj ango.html 

A good amount of this blog is about modern Django. As the author of this blog is also one of 
this book’s authors, the style of the blog loosely resembles the content of this book. 

Django Model Behaviors 

blog.kevinastone.com/dj ango-model-behaviors.html 

Kevin Stone explores how to structure models and associated code in large Django projects. 

Awesome-Django 

awesome-dj ango.com 

A curated list of awesome Django apps, projects and resources. 

Real Pjrthon Blog 

realpython.com/blog/categories/dj ango/ 

In addition to their excellent tutorial book, the Real Python blog contains a lot of useful Django 
material on a wide range of subjects. 


Timeless Beginner Django Material 

Web: 

Django Girls Tutorial 

tutorial.dj angogirIs.org 

Created and maintained by the international Django Girls organization, this is an excellent 
resource no matter your gender. 
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Official Django 1.11 Documentation 

docs.dj angoproj ect.com/en/1.11/ 

The official Django documentation is incredibly useful. If youve used a previous version of 
Django, make sure that you are reading the correct edition of the documentation. 


Timeless Beginner Python Material 

Learn Pj^hon the Hard Way Online Edition 

learnpythonthehardway.org 

By going right to the source, this free for HTML, paid for video resources, is one of the best 
places to start. The video resources are especiaUy useful. 

Automate the Boring Stuff with Python 

amazon.com/gp/product/1593275994 

This fascinating book teaches Python by instructing on how to make boring computer tasks 
easy through automation. Why update 150 columns of a spreadsheet when Python can do it 
for you? 


Timeless Useful Python Material 

Fluent Python 

amzn.to/2oHT0Ra 

One of our favorite Python books, author Luciano Ramalho tours Pythons core language 
features and libraries, and shows us how to make code shorter, faster, and more readable at the 
same time. 

Effective Pjrthon 

amzn.to/lNsiqVr 

Instructs on many useful practices and techniques when coding in Python. 

Python Cookhook, 3rd Edition 

amzn.to/I3Sv6q 

An incredible book by Python luminaries David Beazley and Brian Jones, it’s filled with deli- 
cious ice cream recipes... err... incredibly useful Python recipes for any developer using Python 
3.3 or greater. 

Treading on Python Volume 2 

amzn.to/lkVWi2a 

Covers more advanced Python structures. 
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Writing Idiomatic Python 3.3 

amzn.to/laS5df4 

Great tips for optimizing your code and increasing the legibility of your work. There are a few 
places it differs from our practices (imports being the largest area of difference), but overaU we 
concur. A 2.7 version is available at amzn. to/lf j9j7z 


JavaScript Resources 

Books: 

Secrets of a JavaScript Ninja (Print and Kindle) 
amzn.to/18QzT0r 

Definitive Guide to JavaScript (Print and Kindle) 
amzn.to/lcGVkDD 

JavaScript: The Good Parts (Print and Kindle) 
amzn.to/lauw36x 

JavaScript Patterns (Print and Kindle) 
amzn.to/ldii9Th 

Web Resources: 

Mozdla Developer NetWork 

developer. mozi lla. org/en-LIS/docs/Web/3avaScri pt 
Learning JavaScript Design Patterns 

addyosmani.com/resources/essentialj sdesignpatterns/book/ 

Stack Overflow 

stackoverflow.com/questions/tagged/javascri pt 


WARNING: StayAwayFrom W3Schools 


One problem about JavaScript (and CSS) research on the web is that W3Schools wiU turn 
up at the top of search engine results. This is unfortunate, because much of the data there is 
outdated enough to be incorrect. Be smart and avoid this resource. 

We scan the results page for the MoziUa Developer NetWork (MDN) link, usually around 
the third position, and click on that one. 


470 








Appendix D: Internationalization and 
Localization 


Django and Python provides a lot of very useful tools for dealing with internationalization, local¬ 
ization, and of course, Unicode. 

This appendix, added as of the second edition, contains a list of things helpful for preparing your 
Django application for non-English readers and non-USA users. This list is by no means complete, 
and we invite the reader to provide additional feedback. 


Start Early 

It is always easier to start with and grow an internationalized, localized project than to convert an 
existing project. 


For Python 2.7: Define Python Source Code Encodings 

In PEP 263 we are given a formal specification for defining how encoding of Python modules is 
to occur. Amongst other things, this affects how Python handles Unicode literals. To define this 
encoding in internationalized projects, at the top each module add: 



Or as shown in the next code example: 
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More information can be found at python . org/dev/peps/pep-0263/ 


Wrap Content Strings with Translation Functions 

Every string presented to end users should be wrapped in a translation function. This is de- 
scribed in-depth in the oflicial Django documentation on d j ango. uti Is. translation at docs. 
dj angoproj ect. com/en/1.11/topi cs/i 18n/translation/. Since that is a lot of text to 
swallow, the table on the following page is a reference guide for knowing when and where to use 
what translation function for what tasks. 


Function Purpose Link 


ugettext () For content executed at runtime, e.g. docs. dj angoproj ect. com/ 

form validation errors. en/1.11/topi cs/i 18n/ 

translation/ 

#standard-translation 

ugettext_lazy () For content executed at compile time, docs. dj angoproj ect. com/ 
e.g. verbose_name in models. en/1.11/topi cs/i 18n/ 

translation/ 

#lazy-translation 

docs.dj angoproj ect.com/ 
en/1.11/topics/i18n/ 
translation/ 

#localized-names-of-languages 
Table 35.1: django.utils.translation Function Reference 


string_concat () Replaces the Standard str. join() 
method for joining strings. Rarely 
used. 
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Convention: Use the Underscore Alias to Save Typing 

As you know, normally we arent fans of abbreviations or shortcuts. However, in the case of interna- 
tionalizing Python code, the existing convention is to use a or underscore, to save on letters. 



Don’t Interpolate Words in Sentences 

The golden rule is always have as much grammar as possible in the string, don’t let the 
code piece the grammar together; and generally verbs are the most problematic. 

- Patrick McLoughlan 

We used to construet translation strings all the time, going so far as to include it in the 1.5 edition 
of the book. This is when you use slightly-clever code to construet sentences out of various Python 
objects. For reference, this was part of Example 8.7: 


Example 35.12: Our Bad Code from the 1.5 Edition 

# DON'T DO THIS! 

# Skipping the rest of imports for the sake of brevity 
cTass FlavorActionMixin: 

@property 

def action (self): 

msg = '{0} "is missing acti on.'. format (self._class_) 

raise NotlmplementedError(msg) 

def form_valid(self, form): 

msg = 'Flavor {0}!'.format(self.action) 
messages.info(self.request, msg) 
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return super(FlavorActionMixin, self).form_valid(form) 
# Snipping the rest of this module for the sake of brevity 


While seemingly handy in that it makes for a self-maintaining mixin, it is overly elever in we 

eant internationalize the resuit of ealling self._class _. In other words, you eant just add 

dj ango. uti Is. translati on the foUowing and expeet it to produee anything meaningful for 
translators to work from: 


Example 35.13: Making Things Impossible For Translators 

# DON'T DO THIS! 

from django.utils.translations import ugettext as 

# Skipping the rest of this module for the sake of brevity 

def form_valid(self, form): 

# This generates a useless translation object. 
msg = _('Flavor {0}!'.format(self.action)) 
messages.info(self.request, msg) 

return super(FlavorActionMixin, self).form_valid(form) 

# Skipping the rest of this module for the sake of brevity 

h_ t 


Rather than writing eode that eonstruets sentenees out of various Python eonstruets, now we write 
more meaningful dialogues that ean he readily translated. This means a little more work, hut the 
resuit is a more easily translatable projeet. Henee why we now foUow this pattern: 
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def success_msg(self) : 

return Notimplemented 

class FlavorCreateView( LoginRequiredMixin, FlavorActionMixin, 

CreateView): 

model = Flavor 

# Slightly longer but more meaningful dialogue 
success_msg = _(' Flavor created!') 

# Skipping the rest of this module for the sake of brevity 


For reference, you can combine individual strings representing meaningful sentences and dialogues 
into larger values. However, you shouldnt build sentences by concatenating pieces, because other 
languages may require a different order. For the same reason, you should always include punctuation 
in translated strings. See as follows: 



475 










Chapter 3S: Appendix D: Internationalization and Localization 


Unicode Tricks 

Here are some things weVe learned when dealing with unicode-related issues. 


Python 3 Makes Unicode Easier 

In our experience Python 3 makes Unicode handling much, much easier. While in theory things can 
and are hack-ported to Python 2.7, weVe found that when using Python 3 we just don’t have the 
same kinds of problems. If working on a new project, this is as good a reason as any to considet 
switching to Python 3. 

Yes, there have been articles stating that Python 3 handles Unicode poorly, but those were mostly 
written by someone working at a lower level than 99% of Django developers. See our take on Armin 
Ronachers commentary in Appendix F: Is Moving to Python 3 Worth It? 


Use django.utils.encoding,force_text() Instead of str() 

When you are working with Python 3.x (or Python 2.7) and need to ensure that a useful string-type 
value is returned, don’t use the str () (or uni code () if on Python 2.7) built-ins. What can happen 
is that under certain circumstances, instead of returning a Unicode or str object, Django will return 

a nigh-meaningless dj ango. uti Is. functi onal. proxy object, which is a lazy instance of 

the data requested. 

Instead, do as our friend Douglas Miranda suggested to us, and use 

dj ango. uti Is. encodi ng. force_text. In the case that you are dealing with a proxy object or 
lazy instance, it resolves them as strings. 


TIP: Django is Lazy 


One of the ways that Django does optimizations is via lazy loading, a design pattern which 
defers initialization of an object untU it is needed. The place where this is most obviously 
used is Djangos ORM, as described at docs.djangoproject.com/en/1.11/topics/ 
db/queries/#querysets-are-lazy. This use of lazy objects can cause problems with 
display of content, hence the need for django. uti Is. encodi ng. force_text (). 
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Browser Page Layout 


Assuming youVe got your content and Django templates internationalized and localized, you can 
discover that your layouts are broken. 

A good Django-based example is Mozilla and their various sites for supporting tools like Firefox. On 
these sites they handle translations for over 80 languages. Unfortunately, a title that fits the page in 
English breaks the site in more verbose languages such as German. 

Mozilla’s answer is to determine the width of a title Container, then use JavaScript adjust the font 
size of the title text downwards until the text fits into the Container with wrapping. 

A simpler way of handling this issue is to assume that other languages can take up twice as much 
space as English. English is a pretty concise language that, because of its short words, handles text 
wrapping very well. 


The ChaUenges of Time 

Time is a tricky subject even without timezones and different calendering systems. Here are resources 
to help navigate through time-based coding: 

>• yourcalendricalfallacyis.com 
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Appendix E: Settings Alternatives 


Here a couple of alternative patterns for managing settings that we feel can be recommended. They 
avoid the local_settings anti-pattern and allow for management of configuration that willworkwith 
either the Environment Variables Pattern or the Secrets File Pattern. 


WARNING: Converting Existing Settings is Hard 


If you have an existing project using multiple settings modules and you want to convert it 
to the single settings style, you might want to reconsider. Migrating settings approaches is 
always a tricky process, and requires deep and wide test coverage. Even with the best test 
coverage, there is a chance it’s not going to be worth it. 

For these reasons, we suggest being conservative about switching to new settings approaches. 
Only do it when the current settings management approach has become a pain point, not 
when a new method becomes popular. 


Twelve Factor-Style Settings 

If we’re relying on environment variables, why not use the simplest settings.py system possible? Bruno 
Renie, creator of django-floppyforms and FeedHQ_(feedhq . org), advocates an alternate approach 
to Django settings files, in which ali environments use the same single settings file. 

The argument for this approach is that when using the multiple settings files approach, you end up 
with environment-specific code. For instance, when doinglocal development, youre not running the 
code with production settings. This increases the chance of running into production-specific bugs 
when you update some code without updating the production settings accordingly. 
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This style involves using sensible default settings and as few environment specific values as possible. 
When combined with tools like Vagrant and Docker, it means that mirroring production is trivial. 

It restdts in a much simpler settings file, and for Twelve Factor App fans, it’s right in line with that 
approach. 

If you want to see an example of the approach in action, check out FeedHQls settings module: 
gi thub.com/feedhq/feedhq/blob/master/feedhq/setti ngs .py 

WeVe enjoyed this approach for new and smaller projects. When done right, it makes things elegantly 
simple. 

However, it’s not a perfect solution for aU problems: 

>■ It doesnt provide much benefit for simplification when development environments are drasti- 
cally different than production. 

>- It doesnt work as weU with projects being deployed to more than one operating system. 

>- Complex settings on large projects are not really simplified or shortened by this approach. It 
can be chaUenging to use on large or complex projects. 

If you would like to know more about this approach, we recommend the foUowing articles: 

>• bruno.im/2013/may/18/dj ango-stop-writing-settings-files/ 

>• 12factor.net/config 
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Django 1.11 is the last version of the framework to support Python 2.7. Starting with Django 2.0 
you must use Python 3.5 or higher. Fortunately, Django 1.11 is a long-term support (LTS) version, 
and will be maintained by the Django core team until 2020. 

This should provide enough time to migrate a project from Python 2.7. Since the common critical 
Python libraries all support versions of Python 3, not moving upwards is often a factor of inertia 
rather than a lack of library conversion. 


Is Moving to Python 3 Worth It? 

Ifyou plan to continue using Django after 1.11, then moving to Python 3 isn’t worth it, it’s a necessity. 

Since the autumn of 2015 we ve exclusively coded ali of our Django projects in Python 3.4 or higher. 
For us, the dilferences between Python 3 and 2 were minor, including when it came to us converting 
some of the third-party libraries we maintain. 

One of the features we like most of all about Python 3 is how easy it makes working with Unicode. 
No longer are we mystified by Python 2’s inconsistent behavior. 


TIP: But Armin said Python 3’s Unicode Handling Sucks! 


Armin Ronacher, creator of Jinja2, Flask, Click, and other libraries, is notable for his critiques 
of how Python 3 handles Unicode. Flis arguments were valid for him: his work often touches 
low-level features of Python that are not the normal thing web developers encounter on 
a day-to-day basis. These critiques spurred the Python core team to address some of the 
shortcomings he identified, and other issues have been resolved by working around the quirks 
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of Python 3. 

The end resuit of Armihs critiques is that in the world of Python 3, it’s rare to encounter a 
Unicode prohlem. While thats not to say everything is fixed, let’s just say that as US-educated 
coders we’re no longer afraid of Unicode. 


Django’s Documentation on Python 3 

Djangos ofhcial documentation has a weU-written page that covers much of whats involved with 
workingwith Python 3. Here are some specific sections you should pay careful attention to: 

Django OfficialDocs onP 3 rthon3 docs.djangoproject. com/en/1.11/topi cs/pythonS/ 
Coding Gtiidelines docs. dj angopro j ect. com/en/1.11/topi cs/py thon3/ 

#coding-guidelines 

Writing Compatible Code with Six docs. dj angopro j ect. com/en/1.11/topi cs/ 
python3/#writing-compatible-code-with-six 

Here are some other things you should know. 


Most Critical Packages Work with Python 3 


As of the time of this writing, Django Packages lists over 1232 packages that support Python 3. A 
lot more work with Python 3, just haveht identified them as such. This includes such critical libraries 
as: 


>- Django itself 
>- Pillow 
>- django-allauth 
>- django-braces 
>■ django-crispy-forms 
>- django-debug-toolbar 
>■ django-rest-framework 
>- python-requests 


You can see a list of Django specific libraries at dj angopackages. org/python3/. 
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Checking for Python 3 Compatibility 

Here are the steps we follow in rough order when determining if a third-party library actually works 
with Python 3: 

>- Checkon https://www.di angopackages.org/pvthon3/ . 

>• Look up the package on PyPI and see if any of its trove classifiers mention Python 3 status. 

>- See if a puU request for Python 3 support is outstanding. 

>- Run the test suite using Python 3.5 or higher. 

>- If a Django project, check the models for_ str _() methods. If it has them, it’s a pretty 

good indicator that it’s Python 3 friendly. 

Converting a Library to Work with Python 3 

How we converted our Python 2 code to Python 3: 

>- Get the test harness working with Python 3. 

>• Lean on dj ango. contri b. si x as much as possihle. Add a compat.py module only if ahso- 
lutely needed. 

>- Fix any prohlems you find in the code. Try to keep Solutions as simple as possible. 

>- Submit the pull request. 

>- Politely poke the package owner to accept the pull request. 

>- Once the owner accepts the pull request, gently poke the package owner to push the update 
to PyPI. 


TIP: Deahngwith Slow Maintainers 


Ranting to or complaining about slow-moving maintainers is absolutely counter-productive. 
People have lives and jobs that sometimes get in the way of open source. It’s more productive 
to be patient, be polite, and if necessary do an absolutely minimal fork or find a working 
alternative. 


Use Python 3.5 or Later 

Django is a large, complicated System. While it’s heavily tested for multiple versions of Python 3, 
weVe found that it just works better with more recent versions of the language. The classic example 
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that often occured was how mi grate failed in curious ways when using Python 3.3.0. 


Working With Python 2 and 3 

We usually encounter this scenario when we are writing a third-party package for use in Django 
or even just vanilla Python. However, there are use cases where an entire Django project might be 
deployed to Python 2.7 as well as Python 3. Fortunately, most of the following suggestions apply no 
matter the scale of the project. 


Tests and Continuous Integration 

If there isht a working test harness and fiinctioning continuous integration, now is the time to set it 
up. Testing compatibility across major Python versions simply requires automation. 


Keep Compatibility Minimally Invasive 

The last thing that a project needs is complex branches to deal with different versions of Python. 
Therefore, use the following imports at the top of a Python module to keep code identical: 
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When we do need more complexity or any sort of logic, thats when it’s time to create a compat.py 
module. 


TIP: Use ”from_future_import absolute_import” 


Python 3 updates and improves how imports work, and it does this in a 
good way. Fortunately, it’s been back-ported to Python 2.7 via the use of the 

from _ future _ import absolute_import statement. Even if you don’t plan to 

use Python 3, this is a great feature and allows for the explicit relative imports demonstrated 
in the table below. 


Use django.utils.encoding,python_2_unicode_coinpatible in Models 

Rather than write both_ str _() and_ uni code _() methods, use 

dj ango. uti Is. encodi ng. python_2_uni code_compati ble so it only has to be written once. 
See Section 19.3: Viewing String Representations of Objects. 


Resources 

The following are useful resources for Python 3 topics: 

Porting to Pjrthon 3 

http://pvthon3porting.com/ 

Lennart Regebros free HTML or paid e-bookbundle on the subject of moving from Python 
2to3. 

Porting Django apps to Python 3 

http://voutu.be/c3MGvAYYUvY 

This is Jacob Kaplan-Moss’ PyCon US 2013 video on the subject. 

Python Cookhook, 3rd Edition 

http://amzn.to/I3Sv6q 

David Beazley and Brian Jones’ book of handy recipes for Python 3. 

Writing Idiomatic Python 3.3 

http://amzn.to/laS5df4 

Jeff Knupps guide to writing Python 3 code the ‘right’ way. 
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Appendix G: Security 
Settings Reference 


In Django, knowing which setting should be set to what value in development vs production requires 
an unfortunate amount of domain knowledge. This appendix is a reference for better understanding 
bow to configure a Django project for both development and production. 


Setting 

Development 

Production 1 

ALLOWED_HOSTS 

any list 

See subsection 33.2.4 

Cross Site Request Forgery protection 

See next page 

See next page 

DEBUG 

True 

False 

DEBUG_PROPAGATE_EXCEPTIONS 

False 

False 

Email SSL 

See next page 

See next page 

HIDDLEWARE_CLASSES 

Standard 

Add SecurityMiddleware 

SECRET_KEY 

Use cryptographic key 

See section 5.3 

SECURE_CONTENT_TYPE_NOSNIFF 

True 

True 

SECURE_PROXY_SSL_HEADER 

None 

See next page 

SECURE_SSL_HOST 

False 

True 

SESSION_COOKIE_SECURE 

False 

True 

SESSION_SERIALIZER 

See below 

See next page 


Table 35.2: Security Settings Reference 
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Cross Site Request Forgery Protection Settings 

For most cases, the Standard Django defaults for these settings are adequate. This list provides refer- 
ences to edge cases and the CSRF setting documentation that might provide mitigation: 

>- Internet Explorer and CSRF failure: docs.djangoproject.com/en/1.11/ref/ 
settings/#csrf-cookie-age 

>- Cross-subdomain request exclusion, (e.g. posting from vanilla.twoscoopspress.com 
to chocolate.twoscoopspress.com): docs.dj angoproj ect.com/en/1.11/ref/ 
settings/#csrf-cookie-domain 

>- Changing the default CSRF failure view: docs. dj angoproj ect. com/en/1.11/ref / 
settings/#csrf-failure-view 


Email SSL 

Django now supports secure connections to SMTP servers. If emails from a site contains security- 
related material, we strongly suggest using this feature. Documentation on the foUowing settings 
begins at docs.dj angoproj ect.com/en/1.11/ref/settings/#emai1-use-tls 

► EMAIL_USE_TLS 

► EMAIL_USE_SSL 

► EMAIL_SSL_CERTFILE 

► EMAIL_SSL_KEYFILE 

SESSION_SERIALIZER 


Per subsection 26.10.4: 

SESSION_SERIALIZER = django. contri b. sessi ons .serializers. ilSONSeri ali zer. 


SECURE_PROXY_SSL_HEADER 


For some setups, most notably Heroku, this should be: 

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') 
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Have a Plan Ready for When Things Go Wrong 

Handling security failures is incredibly stressful. There is a sense of urgency and panic that can over- 
whehn our better judgement, leading to snap decisions that can involve iU-advised ‘bug fixes’ and 
public statements that worsen the problem. 

Therefore, it’s critical that a point-by-point plan be written and made available to maintainers and 
even non-technical participants of a project. Here is a sample plan: 

O Shut everything down or put it in read-only mode. 

0 Put up a static HTML page. 

© Back everything up. 

O After reading docs.dj angoproject. com/en/dev/i nternals/security/, email secu- 
rity@djangoproject.com about your security-related problem, even if it’s your fault. 

0 Start looking into the problem. 

Let’s go over these steps: 


Shut Everything Down or Put It in Read-Only Mode 

The first thing to do is remove the ability for the security problem to continue. That way, further 
damage is hopefuUy prevented. 

On Heroku: 
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For projects you deploy yourself or with automated tools, youre going to have create this capability 
yourself. Fortunately, other people have faced this hefore so we come prepared with reference material: 

>- cyberciti .biz/faq/custom-nginx-maintenance-page-with-http503 forputting 
up maintenance 503 pages. 

>- gi thub. com/shanx/d j ango-mai ntenancemode allows you to turn a site on and off with 
a management command. 

>■ github.com/alsoicode/django-maintenancemode-2 is a database-powered solution 
great for flipping portions of a site in and out of read-only mode. 

>• Other tools can be found at dj angopackages. org/grids/g/emergency-management 


Put Up a Static HTML Page 

You should have a maintenance page formatted and ready to go when you launch your project. This 
way, when things go wrong and youVe shut everything down, you can display that to the end user. If 
done well, the users might understand and give you the breathing room to work out the problem. 


Back Everything Up 

Get a copy of the code and then the data off the servers and keep it on a local hard drive or SSD. You 
might also consider a bonded, professional storage company. 

Why? First, when you back things up at this stage, you are protecting your audit trail. This might 
provide you with the capability to determine where and when things went wrong. 

Second, and this might be unpleasant to hear, but malignant staff can cause as many problems as 
any bug or successful penetration. What that means is that the best software-based security is useless 
against a developer who creates a backdoor or a non-technical stalf level user who decides to cause 
trouble. 
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Email security@djangoproject.com, Even if It’s Your Fault 

As long as your problem is security related, read through docs.djangoproject.com/en/dev/ 
internals/security/, then send a quick email summarizing the problem. Ask for help while 
you are at it. 

There are a number of reasons why this is important: 

>- Writing up a quick summary wiU help you focus and gather your thoughts. Youre going to 
be under an amazing amount of stress. The stress and urgency of the situation can make you 
attempt stupid things that can aggravate the problem. 

>- You never know, the Django security team might have good advice or even an answer for you. 

>- It might be a Django problem! If that is the case, the Django security team needs to know so 
they can mitigate the problem for everyone else before it becomes public. 


TIP: Jacob Kaplan-Moss on Reporting to the Django Project 


Former Django BDFL and former Director of Security for Fleroku Jacob Kaplan-Moss 
says, “Fd much rather have people send things that arent actual problems in Django to secu¬ 
rity# djangoproject.com than accidentaUy disclose security issues publicly because they dont 
know better.” 


Start Looking Into the Problem 

YouVe shut things down, backed everything up, are displaying a static HTML page, emailed secu- 
rity#dj angoproject.com, and are looking at the problem. By foUowing the above steps, youVe given 
yourself (and possibly your team) time to breathe and figure out what really happened. 

This will be a stressful time and people will be on the edge of panic. Start doing research, perhaps in 
this book, ask questions as per Chapter 34: Where and How to Ask Django Questions, and find a 
resolution. 

Before you implement a correction, it’s often better to make sure you have a real, proper fix for the 
problem then do a rushed emergency pateh that destroys everything. Yes, this is where tests and 
continuous integration shine. 
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Staypositive-, now is the time for everyone to come together and fix the problem. Start taking notes, 
ask for help from the best people you know, remind yourself (or the team) that you have the will and 
the smarts to fix things, and make things right! 


WARNING: The Nightmare of the Zero-Day Attack 


Zero-day attacks are attacks on vidnerabilities that are known to the public or a closed list 
but stili unpatched. For example: 

>- Attacks the day of a Software update release, right before most people get around to 
instaUing the Software update. 

>- Attacks right after a user writes up a blog post about a vulnerability they found, before 
package maintainers have the chance to write and release a patch. 

>■ Attacks targeting vulnerabilities in discussion on a closed security mailing list. 

With zero-day attacks, there is often no time to address and patch the vulnerability, making 
the compromise especially difficult to manage. If there was ever a reason to have a battle plan 
for handling security issues, this is it. 

See en.wikipedi a.org/wiki/0day 
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Appendix I: WebSockets with Channels 


Already mentioned in Chapter25: Asynchronous Task Queues, Channels provides the capability 
for Django to handle WebSockets. The advantage of using Channels for this purpose is that unlike 
alternative approaches such as Tornado or Node.js , Channels allows us to use Websockets in a way 
very similar to views. Better yet, by using Channels, we can access our projects code base, allowing 
us to access our models and other custom code. We ve found Channels to be a very powerful tool, 
especially when we follow the same practices we espouse in the rest of this book. 

Keep in mind that the advantage of WebSockets is much more than providing a real-time interface. 
It’s that the protocol is lighter than HTTP. This makes it a faster way to transmit data back and forth 
between client and server. 

Here are our thoughts on using Channels based offlessons learned: 


Each Browser Tab Has Its Own WebSocket Connection 

If a User has one hundred tabs open to our project, then they are connecting to us with one hundred 
WebSockets. As can be imagined, this causes browsers to crash and if enough users share this behavior, 
can overload servers. While we can’t realisticaBy force users to close their tabs, we can optimize servers 
on our end. We cover this in Chapter 24: Finding and Reducing Bottlenecks. 

Another option is to track the number of WebSockets a particular user has to your system. If they 
have more than twenty (or any number of your choosing) open, then close the connections that dont 
seem to be doing anything. We found this to be a pretty effective way of protecting servers from 
unnecessary load. Unfortunately, there isnt yet a stock solution for resolving things in this manner. 
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Expect WebSocket Connections to Drop All the Time 

WebSockets in Channels works very differently than the typical Django request-response cycle. In- 
stead of receiving an HTTP request from a user and then sending out an HTTP response in the 
form of HTML or JSON, WebSockets open a constant connection, or socket, between the server 
and the browser. Hence the term ‘WebSocket’. 

The problem is that the odds are stacked against the connections survival. Let’s go over some of what 
threatens it, shall we? 

>- Small amounts of latency between the browser and the server 

>- The server throws the equivelant of the 500 error 

>- The server crashes 

>- The browser tab crashing 

>- The browser crashing 

>- The user putting their computer to sleep 

Rather than address this problem with some kind of long poli fallback like socket .io or SockilS, 
Django Channels just lets the connection die. When this happens, the client has to trigger a new 
connection. Conveniently, Django Channels provides a small JavaScript library that creates a new 
connection when the old one dies. 

The important thing in all this is to remember is that when using Channels we have to take into 
account that long polling isnt an option we can use. 


Channels Works Better With Python 3.6+ 

Our experience is that WebSockets is more stable on Python 3.6. We dont have evidence to back up 
this claim, but it makes a bit of sense when we think about it. The asynci o Standard library package 
powering Channels needed time to mature. By Python 3.6 it had truly arrived. 

What we’re excited by is the hope that at some point Channels will support the uvloop library. 
For reference, uvloop is a drop-in replacement for asynci o that provides immense speed boosts, 
putting Django in the realm of Node.js performance—or better. 
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Validate Incoming Data! 


WebSockets provide another means for our users to send data to our project. As always, validate in¬ 
coming data before you do anything else with it. Weve used both Django Forms and Django REST 
FrameworkSerializers in this capacity. We cover the former technique in depth in Section 12.1: Val¬ 
idate AU Incoming Data With Django Forms. 


Watch Out For Spaghetti Code 

Alright, it’s time for an admission: Our first real elfort with Channels turned into a nasty piate of 
spaghetti code. We had a blast putting it together but when we were done we realized our backend 
code was unmaintainable. 

Later we took our time, leaned on the concept of fat models and helper files documented as we 
worked, wrote better tests, and embraced Generic Consumers as if they were Class-Based Views. In 
other words, we practiced what we preach. 

In talking to other coders we discovered we werent the only ones whose first Channels project turned 
out messy. The lesson learned from this is that in the excitement to play with Django Channels, it’s 
easy to make basic mistakes. So do yourself a favor the first time you write Channels code: Remember 
to take the time and use Standard bestpractices. Especially when it comes towriting tests: channels. 
readthedocs.io/en/stable/testing.html 
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